Nov 01 08:29:27.053764 ip-10-0-143-244 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 01 08:29:27.053773 ip-10-0-143-244 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 01 08:29:27.053781 ip-10-0-143-244 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 01 08:29:27.053998 ip-10-0-143-244 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 01 08:29:37.091361 ip-10-0-143-244 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 01 08:29:37.091375 ip-10-0-143-244 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 54992d16ec534852861d2d47bab89ef7 --
Nov 01 08:31:27.625325 ip-10-0-143-244 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 08:31:28.036659 ip-10-0-143-244 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:31:28.036659 ip-10-0-143-244 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 08:31:28.036659 ip-10-0-143-244 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:31:28.036659 ip-10-0-143-244 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:31:28.042189 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042061    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:28.042189 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042185    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:28.042189 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042189    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:28.042189 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042192    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:28.042189 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042194    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:28.042189 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042197    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:28.042189 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042200    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042203    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042205    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042210    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042212    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042215    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042218    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042221    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042223    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042226    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042229    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042231    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042234    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042236    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042239    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042241    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042244    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042246    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042249    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042251    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:28.042444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042254    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042257    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042260    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042262    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042265    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042267    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042270    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042272    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042274    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042277    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042280    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042283    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042285    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042288    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042291    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042294    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042296    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042299    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042301    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:28.042939 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042304    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042306    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042309    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042311    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042313    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042316    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042318    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042322    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042327    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042331    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042333    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042336    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042339    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042343    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042348    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042351    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042354    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042357    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042359    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:28.043413 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042362    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042365    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042367    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042370    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042374    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042377    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042379    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042382    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042386    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042388    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042392    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042394    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042397    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042399    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042402    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042405    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042407    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042410    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042412    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042416    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:28.043857 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042419    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042422    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042424    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042427    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042430    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042432    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042435    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042437    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042440    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042442    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042445    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042447    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042450    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042453    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042456    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042458    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042461    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042464    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042467    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:28.044382 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042469    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042472    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.042476    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043392    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043400    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043404    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043407    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043410    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043413    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043415    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043418    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043421    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043423    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043428    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043431    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043434    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043436    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043439    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043442    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043445    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:28.044828 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043447    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043450    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043452    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043455    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043458    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043461    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043464    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043466    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043469    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043472    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043476    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043480    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043483    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043486    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043488    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043491    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043493    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043496    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043498    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:28.045332 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043501    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043503    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043506    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043510    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043513    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043516    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043518    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043521    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043524    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043527    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043529    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043532    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043535    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043537    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043540    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043542    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043545    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043548    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043550    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:28.045794 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043554    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043556    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043559    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043561    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043564    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043567    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043570    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043572    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043575    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043577    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043580    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043583    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043585    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043588    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043591    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043593    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043595    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043598    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043600    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:28.046267 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043603    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043605    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043609    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043611    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043614    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043616    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043619    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043621    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043624    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043626    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043632    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043635    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043637    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043640    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043642    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043645    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043648    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043650    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043653    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043655    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:28.046753 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043658    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043661    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043663    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043667    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043669    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043671    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043674    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043676    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043678    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043681    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043683    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.043686    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044212    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044218    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044222    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044225    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044228    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044230    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044233    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044235    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:28.047230 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044238    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044241    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044245    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044248    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044250    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044269    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044273    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044276    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044278    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044281    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044284    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044287    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044289    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044292    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044294    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044297    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044299    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044302    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044304    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044306    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:28.047708 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044309    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044312    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044314    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044318    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044321    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044324    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044326    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044330    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044332    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044335    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044338    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044340    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044343    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044346    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044350    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044352    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044355    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044357    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044360    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044362    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:28.048182 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044365    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044367    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044370    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044373    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044375    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044378    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044380    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044383    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044386    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044388    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044391    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044393    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044396    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044399    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044401    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044404    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044407    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044409    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044412    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:28.048646 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044414    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044417    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044420    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044422    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044424    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044427    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044429    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044432    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044435    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044437    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044440    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044442    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044445    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044447    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044450    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044452    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044454    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044457    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044460    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044462    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:28.049109 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044465    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044467    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044469    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044472    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044474    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044476    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044479    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044481    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044485    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044489    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044494    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044498    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044501    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044503    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044506    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044509    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044512    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044515    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.044517    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:28.049582 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045560    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045570    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045576    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045581    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045585    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045588    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045593    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045597    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045600    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045603    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045607    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045610    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045613    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045616    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045620    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045623    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045626    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045628    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045634    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045637    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045640    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045643    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045647    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045651    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 08:31:28.050043 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045654    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045657    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045660    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045663    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045670    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045673    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045676    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045679    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045683    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045686    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045689    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045693    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045695    2579 flags.go:64] FLAG: --enable-server="true"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045698    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045702    2579 flags.go:64] FLAG: --event-burst="100"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045706    2579 flags.go:64] FLAG: --event-qps="50"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045708    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045711    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045714    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045718    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045721    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045724    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045727    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045730    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045733    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 08:31:28.050590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045736    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045739    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045742    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045745    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045748    2579 flags.go:64] FLAG: --feature-gates=""
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045752    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045755    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045758    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045761    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045764    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045767    2579 flags.go:64] FLAG: --help="false"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045770    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045774    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045778    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045780    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045784    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045787    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045790    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045793    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045796    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045799    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045802    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045805    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045808    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 01 08:31:28.051235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045811    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045814    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045817    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045820    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045822    2579 flags.go:64] FLAG: --lock-file=""
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045825    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045828    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045831    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045838    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045841    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045844    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045847    2579 flags.go:64] FLAG: --logging-format="text"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045850    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045854    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045857    2579 flags.go:64] FLAG: --manifest-url=""
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045859    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045864    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045867    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045873    2579 flags.go:64] FLAG: --max-pods="110"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045876    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045879    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045883    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045886    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045891    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045894    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 08:31:28.051910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045897    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045904    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045907    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045911    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045914    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045917    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045920    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045923    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045925    2579 flags.go:64] FLAG: --port="10250"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045928    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045931    2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f1abf34e2979c1a9"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045934    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045937    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045940    2579 flags.go:64] FLAG: --register-node="true"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045943    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045948    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045951    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045954    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045957    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045961    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045964    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045967    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045969    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045972    2579 flags.go:64] FLAG: --runonce="false"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045975    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045978    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045981    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 08:31:28.052539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045984    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045987    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045990    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045995    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.045998    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046000    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046003    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046006    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046009    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046011    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046014    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046017    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046040    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046044    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046047    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046055    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046058    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046060    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046063    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046066    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046070    2579 flags.go:64] FLAG: --v="2"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046075    2579 flags.go:64] FLAG: --version="false"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046079    2579 flags.go:64] FLAG: --vmodule=""
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046083    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046086    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 08:31:28.053219 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046228    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046232    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046235    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046238    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046241    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046244    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046247    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046249    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046252    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046255    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046259    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046262    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046264    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046267    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046270    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046273    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046276    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046279    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046282    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046285    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:28.053809 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046288    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046290    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046293    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046295    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046298    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046301    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046303    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046307    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046310    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046312    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046315    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046318    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046320    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046323    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046325    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046328    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046331    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046333    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046336    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:28.054306 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046338    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046341    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046343    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046347    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046350    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046352    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046355    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046358    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046360    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046363    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046365    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046368    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046370    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046373    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046375    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046378    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046380    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046383    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046387    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:28.054755 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046390    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046395    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046397    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046400    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046403    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046405    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046408    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046411    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046413    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046416    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046418    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046421    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046423    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046427    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046430    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046433    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046437    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046440    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046443    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:28.055211 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046446    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046448    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046450    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046453    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046455    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046458    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046460    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046462    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046465    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046468    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046471    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046473    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046476    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046478    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046482    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046484    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046487    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046490    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046492    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046495    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:28.055705 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046498    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046500    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046503    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046505    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046507    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046510    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046513    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046515    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046517    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.046546    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046583    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046587    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046591    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046594    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046597    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:28.056198 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046601    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046606    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046609    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046612    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046615    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046618    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046621    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046625    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046627    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046630    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046632    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046635    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046639    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046642    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046644    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046647    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046650    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046652    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046655    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046657    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:28.056537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046660    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046662    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046665    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046668    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046670    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046673    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046677    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046679    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046682    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046684    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046687    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046690    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046693    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046695    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046698    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046700    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046702    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046705    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046707    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:28.057005 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046710    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046712    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046715    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046717    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046720    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046724    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046727    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046729    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046732    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046734    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046737    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046739    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046742    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046744    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046747    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046749    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046752    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046755    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046758    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046762    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:28.057463 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046764    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046767    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046770    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046772    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046775    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046777    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046780    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046782    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046785    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046787    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046790    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046792    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046795    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046798    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046800    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046803    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046805    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046809    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046811    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:28.057924 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046814    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046816    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046819    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046822    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046824    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046827    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046829    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046832    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046834    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046837    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046839    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046842    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046845    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046848    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046853    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046856    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046859    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046862    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046865    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:28.058454 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046868    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046871    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046873    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.046876    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.054398    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.054415    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.054431    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.054437    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054492    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054498    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054502    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054505    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054508    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054511    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054514    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054517    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054520    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054523    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054525    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054528    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:28.058908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054530    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054533    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054536    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054539    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054541    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054544    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054547    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054549    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054552    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054554    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054557    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054559    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054562    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054564    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054567    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054570    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054573    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054576    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054579    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:28.059439 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054582    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054584    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054587    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054590    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054592    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054594    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054597    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054599    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054602    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054604    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054607    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054609    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054612    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054614    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054617    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054619    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054622    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054625    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054627    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054630    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:28.059878 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054632    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054635    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054637    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054640    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054642    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054646    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054650    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054653    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054656    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054659    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054662    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054665    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054668    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054671    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054673    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054676    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054678    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054680    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054683    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:28.060355 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054686    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054688    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054691    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054693    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054696    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054698    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054700    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054703    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054705    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054708    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054711    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054713    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054716    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054718    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054721    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054723    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054726    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054728    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054731    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054733    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:28.060827 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054736    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054738    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054741    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054744    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054747    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054750    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054752    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054755    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054758    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054761    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054763    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054766    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054768    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054771    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054773    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054776    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:28.061345 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.054803    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054942    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054946    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054949    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054952    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054955    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054958    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054960    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054963    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054966    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054968    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054971    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054974    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054976    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054979    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054981    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054984    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054986    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054989    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:28.061863 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054992    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054994    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.054997    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055000    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055002    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055005    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055007    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055010    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055012    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055015    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055017    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055035    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055040    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055043    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055047    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055049    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055052    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055054    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055057    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:28.062537 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055059    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055062    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055066    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055070    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055073    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055076    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055078    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055081    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055084    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055088    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055091    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055094    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055096    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055100    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055102    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055105    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055107    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055110    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055113    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:28.063185 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055116    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055118    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055121    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055123    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055126    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055128    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055131    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055133    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055136    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055138    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055141    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055143    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055146    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055148    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055151    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055153    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055156    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055158    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055161    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055163    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:28.063812 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055166    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055169    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055171    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055174    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055176    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055179    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055182    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055184    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055187    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055189    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055191    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055195    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055198    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055201    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055203    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055206    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055208    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055211    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055213    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055216    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:28.064502 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055219    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055221    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055224    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055227    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055229    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055232    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055234    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055237    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055239    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055241    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.055264    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055296    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055300    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055304    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055306    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055309    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:31:28.065193 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055311    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055314    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055317    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055319    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055329    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055333    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055335    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055338    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055342    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055345    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055348    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055350    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055353    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055355    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055358    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055360    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055363    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055365    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055368    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055370    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:31:28.065579 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055373    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055375    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055378    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055381    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055383    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055386    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055389    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055391    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055394    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055396    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055399    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055401    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055404    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055406    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055409    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055411    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055414    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055416    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055419    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:31:28.066047 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055421    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055423    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055426    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055429    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055432    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055437    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055440    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055442    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055445    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055447    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055449    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055452    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055456    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055460    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055464    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055466    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055469    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055473    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:31:28.066655 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055477    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055480    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055483    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055486    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055488    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055491    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055493    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055496    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055498    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055501    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055503    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055506    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055508    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055511    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055513    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055516    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055518    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055521    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055524    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055528    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:31:28.067261 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055530    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055533    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055535    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055538    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055540    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055543    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055546    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055548    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055551    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055553    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055556    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055560    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055563    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055566    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055568    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055571    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055573    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055576    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055578    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055581    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:31:28.067738 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055583    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055585    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055588    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:28.055590    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.055648    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.058095    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.059096    2579 server.go:1014] "Starting client certificate rotation"
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.059189    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.059224    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.061164    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 08:31:28.068283 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.066242    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 08:31:28.081221 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.081186    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 01 08:31:28.084977 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.084961    2579 log.go:25] "Validated CRI v1 image API"
Nov 01 08:31:28.085843 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.085815    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 08:31:28.090298 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.090276    2579 fs.go:135] Filesystem UUIDs: map[0f5fda05-0f32-4c8e-a4ba-e1a001e678c3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a9b872ad-a124-47a2-adba-9decb421333f:/dev/nvme0n1p3]
Nov 01 08:31:28.090362 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.090298    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 08:31:28.095815 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.095693    2579 manager.go:217] Machine: {Timestamp:2026-11-01 08:31:28.094352933 +0000 UTC m=+0.361756442 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3138317 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f054e35028055c95c233acd14d966 SystemUUID:ec2f054e-3502-8055-c95c-233acd14d966 BootID:54992d16-ec53-4852-861d-2d47bab89ef7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c6:16:03:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c6:16:03:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:25:ce:f2:3f:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 08:31:28.095815 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.095814    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 08:31:28.095953 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.095937    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 08:31:28.098607 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.098581    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 08:31:28.098749 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.098609    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-244.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 08:31:28.098815 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.098756    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 08:31:28.098815 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.098764    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 08:31:28.098815 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.098805    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 08:31:28.098887 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.098823    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 08:31:28.099809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.099797    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 08:31:28.104190 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.104177    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 08:31:28.104315 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.104306    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 08:31:28.105105 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.105095    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 08:31:28.105160 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.105110    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 08:31:28.105160 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.105124    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 08:31:28.105160 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.105135    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 01 08:31:28.105277 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.105195    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 08:31:28.105573 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.105553    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 08:31:28.107941 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.107919    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 08:31:28.116916 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.116891    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 08:31:28.118423 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.118398    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 08:31:28.118512 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.118446    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 08:31:28.118710 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.118684    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 08:31:28.118710 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.118710    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 08:31:28.118813 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.118743    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 08:31:28.118937 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.118927    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 08:31:28.118977 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.118942    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 08:31:28.118977 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.118952    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 08:31:28.119369 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.119359    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 08:31:28.119419 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.119372    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 08:31:28.119419 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.119386    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 08:31:28.119419 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.119396    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 08:31:28.119419 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.119409    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 08:31:28.119626 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.119424    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 08:31:28.120977 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.120965    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 08:31:28.121066 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.120985    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 08:31:28.124948 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.124924    2579 server.go:1263] "Started kubelet"
Nov 01 08:31:28.125092 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.125006    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 08:31:28.125163 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.125045    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 08:31:28.125203 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.125191    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 08:31:28.125804 ip-10-0-143-244 systemd[1]: Started Kubernetes Kubelet.
Nov 01 08:31:28.126891 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.126820    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 08:31:28.127773 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.127761    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 08:31:28.131552 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.131535    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 08:31:28.133785 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.133763    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 08:31:28.134397 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.134382    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 08:31:28.134957 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.134615    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.us-west-2.compute.internal\" not found"
Nov 01 08:31:28.135178 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135160    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 08:31:28.135252 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135199    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 08:31:28.135252 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135211    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 08:31:28.135351 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135301    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 08:31:28.135351 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135316    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 08:31:28.135469 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135449    2579 factory.go:153] Registering CRI-O factory
Nov 01 08:31:28.135469 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135469    2579 factory.go:223] Registration of the crio container factory successfully
Nov 01 08:31:28.135624 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135521    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 08:31:28.135624 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135535    2579 factory.go:55] Registering systemd factory
Nov 01 08:31:28.135624 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135540    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 01 08:31:28.135624 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135552    2579 factory.go:103] Registering Raw factory
Nov 01 08:31:28.135624 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.135563    2579 manager.go:1199] Started watching for new ooms in manager
Nov 01 08:31:28.138504 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.138483    2579 manager.go:319] Starting recovery of all containers
Nov 01 08:31:28.152122 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.152106    2579 manager.go:324] Recovery completed
Nov 01 08:31:28.156011 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.155997    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:28.157669 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.157640    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:28.157728 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.157684    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:28.157728 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.157700    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:28.158300 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.158285    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 01 08:31:28.158300 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.158298    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 08:31:28.158423 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.158309    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:28.158423 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.158320    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 08:31:28.158937 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.158925    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:28.158981 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.158951    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:28.158981 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.158964    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:28.160495 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.160483    2579 policy_none.go:50] "Start"
Nov 01 08:31:28.160547 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.160502    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 08:31:28.160547 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.160516    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 08:31:28.162485 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.162471    2579 policy_none.go:44] "Start"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.200618    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.200642    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.202220    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.202244    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.202272    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.202310    2579 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.205427    2579 manager.go:341] "Starting Device Plugin manager"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.205491    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.205504    2579 server.go:87] "Starting device plugin registration server"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.205805    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.205819    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.205928    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.206066    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.206081    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.206660    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 08:31:28.211446 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.206710    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-244.us-west-2.compute.internal\" not found"
Nov 01 08:31:28.259339 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.259295    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-244.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 08:31:28.259339 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.259303    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-244.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 08:31:28.261598 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.259357    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-244.us-west-2.compute.internal.18e3dea79e926d81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-244.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-244.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:28.124890497 +0000 UTC m=+0.392294001,LastTimestamp:2026-11-01 08:31:28.124890497 +0000 UTC m=+0.392294001,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-244.us-west-2.compute.internal,}"
Nov 01 08:31:28.303296 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.303220    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"]
Nov 01 08:31:28.303403 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.303298    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:28.304391 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.304371    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:28.304464 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.304412    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:28.304464 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.304423    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:28.306143 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.306128    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:28.306566 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.306545    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:28.306703 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.306685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.306770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.306758    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:28.306971 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.306949    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:28.307095 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.306994    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:28.307095 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.307010    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:28.307095 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.307054    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.307357 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.307340    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:28.307399 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.307371    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:28.307399 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.307382    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:28.307465 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.307340    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:28.307465 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.307447    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:28.307465 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.307458    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:28.309391 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.309375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.309471 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.309400    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:31:28.310087 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.310070    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:31:28.310164 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.310101    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:31:28.310164 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.310121    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:31:28.321660 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.321641    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 08:31:28.321751 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.321701    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tpc2m"
Nov 01 08:31:28.329206 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.329186    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-244.us-west-2.compute.internal\" not found" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.331116 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.331101    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-244.us-west-2.compute.internal\" not found" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.334136 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.334112    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 01 08:31:28.334274 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.334173    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-244.us-west-2.compute.internal.18e3dea7a0869348  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-244.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-244.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-244.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:28.157668168 +0000 UTC m=+0.425071674,LastTimestamp:2026-11-01 08:31:28.157668168 +0000 UTC m=+0.425071674,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-244.us-west-2.compute.internal,}"
Nov 01 08:31:28.336594 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.336577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.336668 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.336600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.336668 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.336624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36a0826bc10ea7285dbce782929bcb3b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"36a0826bc10ea7285dbce782929bcb3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.410688 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.410651    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:28.412114 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.412097    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:28.436924 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.436893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.437082 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.436927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.437082 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.436944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36a0826bc10ea7285dbce782929bcb3b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"36a0826bc10ea7285dbce782929bcb3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.437082 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.436964    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.437082 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.436981    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36a0826bc10ea7285dbce782929bcb3b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"36a0826bc10ea7285dbce782929bcb3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.437082 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.437044    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.505814 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.505783    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:28.506347 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.506329    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.506422 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:28.506360    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-244.us-west-2.compute.internal\": node \"ip-10-0-143-244.us-west-2.compute.internal\" not found"
Nov 01 08:31:28.534785 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.534757    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.592422 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.592350    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:31:28.593225 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.593213    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.634093 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.634066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.636249 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.636227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:31:28.651330 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:28.651308    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:31:29.059351 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.059264    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 08:31:29.059821 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.059503    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 08:31:29.059821 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.059503    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 08:31:29.059821 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.059487    2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-244.us-west-2.compute.internal.18e3dea7a0869348\": read tcp 10.0.143.244:43260->3.130.67.7:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-244.us-west-2.compute.internal.18e3dea7a0869348  default   4926 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-244.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-244.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-244.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:28 +0000 UTC,LastTimestamp:2026-11-01 08:31:28.306972136 +0000 UTC m=+0.574375643,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-244.us-west-2.compute.internal,}"
Nov 01 08:31:29.105327 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.105296    2579 apiserver.go:52] "Watching apiserver"
Nov 01 08:31:29.134557 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.134524    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 08:31:29.326921 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.326834    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 08:26:28 +0000 UTC" deadline="2028-07-11 02:08:22.520187826 +0000 UTC"
Nov 01 08:31:29.326921 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.326862    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14825h36m53.193328725s"
Nov 01 08:31:29.376595 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.376558    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:29.376839 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.376824    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 08:31:29.377135 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.377119    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-ln7w5","openshift-cluster-node-tuning-operator/tuned-cskq4","openshift-dns/node-resolver-8t24r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh","openshift-image-registry/node-ca-jzchj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal","openshift-multus/multus-2gdwv","openshift-multus/multus-additional-cni-plugins-jnc4r","openshift-network-diagnostics/network-check-target-mnsws","openshift-ovn-kubernetes/ovnkube-node-bj2zg","kube-system/global-pull-secret-syncer-v7gkj","kube-system/konnectivity-agent-vlpgj","kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-nlwtz"]
Nov 01 08:31:29.379527 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.379510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.381769 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.381710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.383791 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.383772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.385862 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.385847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.387924 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.387908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.395164 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.395148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.396962 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.396947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.398580 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.398559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:29.398656 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.398636    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:29.400360 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.400344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.402183 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.402167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.402253 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.402227    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:29.404201 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.404188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:29.405945 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.405925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:29.406038 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.405979    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:29.429702 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.429664    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 08:31:29.429855 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.429745    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fl8bw"
Nov 01 08:31:29.436235 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.436201    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 08:31:29.436644 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.436619    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:29.436749 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.436619    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:29.436801 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.436751    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 08:31:29.436801 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.436759    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xk5lv\""
Nov 01 08:31:29.436914 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.436893    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:29.439545 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.439524    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:29.439843 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.439815    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 08:31:29.439843 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.439816    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-24cps\""
Nov 01 08:31:29.439843 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.439821    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 08:31:29.440018 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.439822    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8d96l\""
Nov 01 08:31:29.440018 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.439913    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 08:31:29.440018 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.439966    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48mlk\""
Nov 01 08:31:29.440133 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.440093    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 08:31:29.440476 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.440463    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 08:31:29.441168 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.441130    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 08:31:29.441265 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.441236    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqp75\""
Nov 01 08:31:29.441265 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.441247    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 08:31:29.441265 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.441252    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 08:31:29.442306 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-bin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.442353 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.442353 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-script-lib\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.442353 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-kubelet-config\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.442470 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-run\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.442470 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-host\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.442470 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.442587 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9jm\" (UniqueName: \"kubernetes.io/projected/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-kube-api-access-qf9jm\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.442587 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-conf\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.442587 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-systemd\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.442587 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-hostroot\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.442587 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-conf-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.442587 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-daemon-config\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.442835 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-kubelet\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.442835 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5d1e97d-be9c-4f45-827c-19f789823a3b-agent-certs\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:29.442835 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89776b0e-b9bc-415b-8138-9ce3e9264a27-host-slash\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.442835 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-sys\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.442835 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx899\" (UniqueName: \"kubernetes.io/projected/a108cdc6-7b79-4e7a-8b7d-7824406258e3-kube-api-access-bx899\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.442835 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e38ced3e-d98c-4df1-935f-a1cdc2abf313-host\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.442835 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-netns\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-systemd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-config\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmk98\" (UniqueName: \"kubernetes.io/projected/608b447e-9dda-4503-bcef-26d6af784cfb-kube-api-access-hmk98\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.442991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89q2k\" (UniqueName: \"kubernetes.io/projected/e38ced3e-d98c-4df1-935f-a1cdc2abf313-kube-api-access-89q2k\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-systemd-units\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-netns\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.443124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-dbus\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-modprobe-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-tuned\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-tmp\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cnibin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-etc-kubernetes\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cnibin\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-env-overrides\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysconfig\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.443455 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.443802 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38ced3e-d98c-4df1-935f-a1cdc2abf313-serviceca\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.443833 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-system-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.443833 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:29.443905 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cni-binary-copy\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.443905 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-kubelet\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.443905 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:29.444047 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-registration-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.444047 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-sys-fs\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.444047 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-etc-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444047 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.443992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-ovn\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444047 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovn-node-metrics-cert\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt52q\" (UniqueName: \"kubernetes.io/projected/67b4458b-d683-447a-a253-5b37d8531843-kube-api-access-bt52q\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.444263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/608b447e-9dda-4503-bcef-26d6af784cfb-hosts-file\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.444263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/608b447e-9dda-4503-bcef-26d6af784cfb-tmp-dir\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.444263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-slash\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-var-lib-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-var-lib-kubelet\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.444263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-socket-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.444263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-socket-dir-parent\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-netd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6997\" (UniqueName: \"kubernetes.io/projected/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-kube-api-access-s6997\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5d1e97d-be9c-4f45-827c-19f789823a3b-konnectivity-ca\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5czf\" (UniqueName: \"kubernetes.io/projected/89776b0e-b9bc-415b-8138-9ce3e9264a27-kube-api-access-q5czf\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5kb\" (UniqueName: \"kubernetes.io/projected/afb159a0-e7ce-43e5-be5f-5260297660cb-kube-api-access-2k5kb\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-kubernetes\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-node-log\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-bin\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.444560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-multus\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.445046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.445046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/89776b0e-b9bc-415b-8138-9ce3e9264a27-iptables-alerter-script\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.445046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-device-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.445046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-multus-certs\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.445046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pcz\" (UniqueName: \"kubernetes.io/projected/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-kube-api-access-66pcz\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.445046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-log-socket\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.445046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-lib-modules\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.445046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.444799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-k8s-cni-cncf-io\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.448710 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.448697    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-l9vpz\""
Nov 01 08:31:29.535993 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.535966    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 08:31:29.536163 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536145    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hnxt\""
Nov 01 08:31:29.536234 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536205    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxggb\""
Nov 01 08:31:29.536288 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536233    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cdn26\""
Nov 01 08:31:29.536288 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536273    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536580    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536598    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536606    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536624    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536646    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536648    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536689    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536695    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536706    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536731    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536748    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 08:31:29.536770 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.536753    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 08:31:29.546076 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6997\" (UniqueName: \"kubernetes.io/projected/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-kube-api-access-s6997\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546175 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.546175 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546330 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.546330 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546330 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5d1e97d-be9c-4f45-827c-19f789823a3b-konnectivity-ca\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:29.546330 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5czf\" (UniqueName: \"kubernetes.io/projected/89776b0e-b9bc-415b-8138-9ce3e9264a27-kube-api-access-q5czf\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.546330 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5kb\" (UniqueName: \"kubernetes.io/projected/afb159a0-e7ce-43e5-be5f-5260297660cb-kube-api-access-2k5kb\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:29.546330 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-kubernetes\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.546594 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.546594 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546594 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546424    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546594 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546484    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-kubernetes\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.546594 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-node-log\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546594 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546550    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-node-log\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-bin\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546604    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-multus\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546647    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-bin\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/89776b0e-b9bc-415b-8138-9ce3e9264a27-iptables-alerter-script\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-device-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546705    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-multus\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-multus-certs\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546750    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pcz\" (UniqueName: \"kubernetes.io/projected/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-kube-api-access-66pcz\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546773    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-multus-certs\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-log-socket\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-device-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5d1e97d-be9c-4f45-827c-19f789823a3b-konnectivity-ca\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546847    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-log-socket\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.546852 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-lib-modules\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-k8s-cni-cncf-io\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-bin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-script-lib\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546974    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-bin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546989    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-k8s-cni-cncf-io\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.546996    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-lib-modules\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-kubelet-config\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-run\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-host\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-kubelet-config\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9jm\" (UniqueName: \"kubernetes.io/projected/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-kube-api-access-qf9jm\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547141    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-run\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-conf\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547261    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.547676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-systemd\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-hostroot\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-conf-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-daemon-config\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547349    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/89776b0e-b9bc-415b-8138-9ce3e9264a27-iptables-alerter-script\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-kubelet\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547436    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-conf\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547637    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-hostroot\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547378    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-systemd\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547182    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-host\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5d1e97d-be9c-4f45-827c-19f789823a3b-agent-certs\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547668    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547678    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-kubelet\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89776b0e-b9bc-415b-8138-9ce3e9264a27-host-slash\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547739    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89776b0e-b9bc-415b-8138-9ce3e9264a27-host-slash\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-sys\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547789    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-sys\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx899\" (UniqueName: \"kubernetes.io/projected/a108cdc6-7b79-4e7a-8b7d-7824406258e3-kube-api-access-bx899\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.548529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547813    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-conf-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e38ced3e-d98c-4df1-935f-a1cdc2abf313-host\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547833    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e38ced3e-d98c-4df1-935f-a1cdc2abf313-host\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-netns\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-systemd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-config\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547959    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-systemd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmk98\" (UniqueName: \"kubernetes.io/projected/608b447e-9dda-4503-bcef-26d6af784cfb-kube-api-access-hmk98\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.547983    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-netns\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89q2k\" (UniqueName: \"kubernetes.io/projected/e38ced3e-d98c-4df1-935f-a1cdc2abf313-kube-api-access-89q2k\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-systemd-units\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-netns\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-dbus\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548204    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-systemd-units\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.549456 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-modprobe-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-tuned\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-tmp\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cnibin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-modprobe-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-etc-kubernetes\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cnibin\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548341    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-netns\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548346    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-env-overrides\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548361    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-script-lib\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysconfig\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548397    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-dbus\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548426    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38ced3e-d98c-4df1-935f-a1cdc2abf313-serviceca\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548457    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysconfig\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.550698 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-system-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cni-binary-copy\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-kubelet\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548605    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548661    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-registration-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.548752    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.548826    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret podName:2e9c65da-47f6-494b-819e-34ee9896ff41 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:30.048790843 +0000 UTC m=+2.316194346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret") pod "global-pull-secret-syncer-v7gkj" (UID: "2e9c65da-47f6-494b-819e-34ee9896ff41") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548954    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-daemon-config\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549483    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-kubelet\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.548611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-registration-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-sys-fs\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-etc-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-ovn\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovn-node-metrics-cert\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549756    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-system-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.551700 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt52q\" (UniqueName: \"kubernetes.io/projected/67b4458b-d683-447a-a253-5b37d8531843-kube-api-access-bt52q\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/608b447e-9dda-4503-bcef-26d6af784cfb-hosts-file\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549870    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-etc-kubernetes\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/608b447e-9dda-4503-bcef-26d6af784cfb-tmp-dir\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549891    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cnibin\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-slash\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-var-lib-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549946    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-etc-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-var-lib-kubelet\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-socket-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-socket-dir-parent\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-netd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550164    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-netd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550164    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38ced3e-d98c-4df1-935f-a1cdc2abf313-serviceca\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550204    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cni-binary-copy\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550223    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/608b447e-9dda-4503-bcef-26d6af784cfb-hosts-file\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550249    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-config\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.552503 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550278    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-var-lib-kubelet\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.549997    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-sys-fs\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550298    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-ovn\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550342    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-socket-dir-parent\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550369    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550386    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cnibin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-slash\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.550449    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.550492    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-var-lib-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.550510    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:31:30.05049482 +0000 UTC m=+2.317898323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.551308    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/608b447e-9dda-4503-bcef-26d6af784cfb-tmp-dir\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.551629    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.551718    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-socket-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.551956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.552076    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-env-overrides\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.552352    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-tuned\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.552416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-tmp\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.553051 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.552654    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5d1e97d-be9c-4f45-827c-19f789823a3b-agent-certs\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:29.553709 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.553120    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovn-node-metrics-cert\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.570423 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.570405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:29.608917 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.608845    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5kb\" (UniqueName: \"kubernetes.io/projected/afb159a0-e7ce-43e5-be5f-5260297660cb-kube-api-access-2k5kb\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:29.609246 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.609225    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pcz\" (UniqueName: \"kubernetes.io/projected/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-kube-api-access-66pcz\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.609569 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.609550    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6997\" (UniqueName: \"kubernetes.io/projected/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-kube-api-access-s6997\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.609722 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.609703    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmk98\" (UniqueName: \"kubernetes.io/projected/608b447e-9dda-4503-bcef-26d6af784cfb-kube-api-access-hmk98\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.612721 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.612578    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:29.612721 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.612607    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:29.612721 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.612623    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q6qpf for pod openshift-network-diagnostics/network-check-target-mnsws: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:29.612721 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.612648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5czf\" (UniqueName: \"kubernetes.io/projected/89776b0e-b9bc-415b-8138-9ce3e9264a27-kube-api-access-q5czf\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.612721 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:29.612689    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf podName:44a34856-dd40-4408-9db2-67d075d9850a nodeName:}" failed. No retries permitted until 2026-11-01 08:31:30.112669931 +0000 UTC m=+2.380073446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q6qpf" (UniqueName: "kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf") pod "network-check-target-mnsws" (UID: "44a34856-dd40-4408-9db2-67d075d9850a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:29.612721 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.612689    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9jm\" (UniqueName: \"kubernetes.io/projected/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-kube-api-access-qf9jm\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.613496 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.613470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx899\" (UniqueName: \"kubernetes.io/projected/a108cdc6-7b79-4e7a-8b7d-7824406258e3-kube-api-access-bx899\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.614200 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.614178    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89q2k\" (UniqueName: \"kubernetes.io/projected/e38ced3e-d98c-4df1-935f-a1cdc2abf313-kube-api-access-89q2k\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.614924 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.614906    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt52q\" (UniqueName: \"kubernetes.io/projected/67b4458b-d683-447a-a253-5b37d8531843-kube-api-access-bt52q\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.708536 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.708491    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0fe841d453cc229ff1f7a2450a3ba4.slice/crio-3172d2f31dd0e27d233dc946cff3e5476f89e0650e43f02213088f214f22afe3 WatchSource:0}: Error finding container 3172d2f31dd0e27d233dc946cff3e5476f89e0650e43f02213088f214f22afe3: Status 404 returned error can't find the container with id 3172d2f31dd0e27d233dc946cff3e5476f89e0650e43f02213088f214f22afe3
Nov 01 08:31:29.708888 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.708861    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a0826bc10ea7285dbce782929bcb3b.slice/crio-b8a9fac23e00f919297a254f0b8e41dbcca48a06164a7536e8eed8ac5d6396fe WatchSource:0}: Error finding container b8a9fac23e00f919297a254f0b8e41dbcca48a06164a7536e8eed8ac5d6396fe: Status 404 returned error can't find the container with id b8a9fac23e00f919297a254f0b8e41dbcca48a06164a7536e8eed8ac5d6396fe
Nov 01 08:31:29.708964 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.708909    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:31:29.717526 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.717505    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:31:29.720144 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.720127    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89776b0e_b9bc_415b_8138_9ce3e9264a27.slice/crio-128be4e192bcc15c75ab99af7509cf411896b091610e9e8cdbeb359d6bcc8db3 WatchSource:0}: Error finding container 128be4e192bcc15c75ab99af7509cf411896b091610e9e8cdbeb359d6bcc8db3: Status 404 returned error can't find the container with id 128be4e192bcc15c75ab99af7509cf411896b091610e9e8cdbeb359d6bcc8db3
Nov 01 08:31:29.745502 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.745453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:31:29.756671 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.756646    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda108cdc6_7b79_4e7a_8b7d_7824406258e3.slice/crio-d30d1b2d42fb3104b8e5524cfca4319a4e20d93519509b578b85d1e976659130 WatchSource:0}: Error finding container d30d1b2d42fb3104b8e5524cfca4319a4e20d93519509b578b85d1e976659130: Status 404 returned error can't find the container with id d30d1b2d42fb3104b8e5524cfca4319a4e20d93519509b578b85d1e976659130
Nov 01 08:31:29.761622 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.761602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:31:29.770441 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.770417    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608b447e_9dda_4503_bcef_26d6af784cfb.slice/crio-7d937ad8cda14b416ef7bdd404925f1b9e587015019b480934b0b9976104762a WatchSource:0}: Error finding container 7d937ad8cda14b416ef7bdd404925f1b9e587015019b480934b0b9976104762a: Status 404 returned error can't find the container with id 7d937ad8cda14b416ef7bdd404925f1b9e587015019b480934b0b9976104762a
Nov 01 08:31:29.787161 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.787134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:31:29.796311 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.796285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:31:29.799158 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.796708    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b4458b_d683_447a_a253_5b37d8531843.slice/crio-d7c66d7176e870ead18d1eb0c2b2428066eefca505fb985b98e772c34faca2cf WatchSource:0}: Error finding container d7c66d7176e870ead18d1eb0c2b2428066eefca505fb985b98e772c34faca2cf: Status 404 returned error can't find the container with id d7c66d7176e870ead18d1eb0c2b2428066eefca505fb985b98e772c34faca2cf
Nov 01 08:31:29.805716 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.805695    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38ced3e_d98c_4df1_935f_a1cdc2abf313.slice/crio-85a833ba2aa0af1bacf87e321ad076a890170df0ea9a2174f893a2c085d25654 WatchSource:0}: Error finding container 85a833ba2aa0af1bacf87e321ad076a890170df0ea9a2174f893a2c085d25654: Status 404 returned error can't find the container with id 85a833ba2aa0af1bacf87e321ad076a890170df0ea9a2174f893a2c085d25654
Nov 01 08:31:29.816272 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.816251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2gdwv"
Nov 01 08:31:29.825539 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.825517    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8bc7e3_ca5d_4db2_8a63_9b48e38b5900.slice/crio-575e2349e4cf1c7c07bc74d785c8bade94d22027956713ebf950878a15305aa0 WatchSource:0}: Error finding container 575e2349e4cf1c7c07bc74d785c8bade94d22027956713ebf950878a15305aa0: Status 404 returned error can't find the container with id 575e2349e4cf1c7c07bc74d785c8bade94d22027956713ebf950878a15305aa0
Nov 01 08:31:29.847626 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.847597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:31:29.857591 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.857569    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a214d76_85af_4fd5_9ccd_6cdce46fdd6e.slice/crio-1a64d543be1b8d2e4710491f63dd11bbaa22a4ab920dd1c3cd5001fa5a48c7cb WatchSource:0}: Error finding container 1a64d543be1b8d2e4710491f63dd11bbaa22a4ab920dd1c3cd5001fa5a48c7cb: Status 404 returned error can't find the container with id 1a64d543be1b8d2e4710491f63dd11bbaa22a4ab920dd1c3cd5001fa5a48c7cb
Nov 01 08:31:29.874252 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:29.874232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:29.884273 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:29.884245    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13e4499_bb2f_4e37_bd37_52f4bdad02dd.slice/crio-8b821ad49a7a7feb12fa1a5446ad465996116eb48e8c1c6f6885d8ebf8a172bd WatchSource:0}: Error finding container 8b821ad49a7a7feb12fa1a5446ad465996116eb48e8c1c6f6885d8ebf8a172bd: Status 404 returned error can't find the container with id 8b821ad49a7a7feb12fa1a5446ad465996116eb48e8c1c6f6885d8ebf8a172bd
Nov 01 08:31:30.018735 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.018701    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:30.054434 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.054389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:30.054434 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.054435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:30.054647 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:30.054546    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:30.054647 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:30.054546    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:30.054647 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:30.054608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret podName:2e9c65da-47f6-494b-819e-34ee9896ff41 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:31.054589524 +0000 UTC m=+3.321993016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret") pod "global-pull-secret-syncer-v7gkj" (UID: "2e9c65da-47f6-494b-819e-34ee9896ff41") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:30.054647 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:30.054633    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:31:31.05462534 +0000 UTC m=+3.322028832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:30.155002 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.154901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:30.155694 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:30.155074    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:30.155694 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:30.155092    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:30.155694 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:30.155103    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q6qpf for pod openshift-network-diagnostics/network-check-target-mnsws: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:30.155694 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:30.155155    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf podName:44a34856-dd40-4408-9db2-67d075d9850a nodeName:}" failed. No retries permitted until 2026-11-01 08:31:31.155141377 +0000 UTC m=+3.422544869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6qpf" (UniqueName: "kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf") pod "network-check-target-mnsws" (UID: "44a34856-dd40-4408-9db2-67d075d9850a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:30.207371 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.207313    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cskq4" event={"ID":"a108cdc6-7b79-4e7a-8b7d-7824406258e3","Type":"ContainerStarted","Data":"d30d1b2d42fb3104b8e5524cfca4319a4e20d93519509b578b85d1e976659130"}
Nov 01 08:31:30.208416 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.208390    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"ca0fe841d453cc229ff1f7a2450a3ba4","Type":"ContainerStarted","Data":"3172d2f31dd0e27d233dc946cff3e5476f89e0650e43f02213088f214f22afe3"}
Nov 01 08:31:30.209504 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.209483    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"36a0826bc10ea7285dbce782929bcb3b","Type":"ContainerStarted","Data":"b8a9fac23e00f919297a254f0b8e41dbcca48a06164a7536e8eed8ac5d6396fe"}
Nov 01 08:31:30.210593 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.210575    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jzchj" event={"ID":"e38ced3e-d98c-4df1-935f-a1cdc2abf313","Type":"ContainerStarted","Data":"85a833ba2aa0af1bacf87e321ad076a890170df0ea9a2174f893a2c085d25654"}
Nov 01 08:31:30.211570 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.211550    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh" event={"ID":"67b4458b-d683-447a-a253-5b37d8531843","Type":"ContainerStarted","Data":"d7c66d7176e870ead18d1eb0c2b2428066eefca505fb985b98e772c34faca2cf"}
Nov 01 08:31:30.212593 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.212575    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ln7w5" event={"ID":"89776b0e-b9bc-415b-8138-9ce3e9264a27","Type":"ContainerStarted","Data":"128be4e192bcc15c75ab99af7509cf411896b091610e9e8cdbeb359d6bcc8db3"}
Nov 01 08:31:30.213709 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.213690    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"8b821ad49a7a7feb12fa1a5446ad465996116eb48e8c1c6f6885d8ebf8a172bd"}
Nov 01 08:31:30.214794 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.214773    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerStarted","Data":"1a64d543be1b8d2e4710491f63dd11bbaa22a4ab920dd1c3cd5001fa5a48c7cb"}
Nov 01 08:31:30.215813 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.215793    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2gdwv" event={"ID":"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900","Type":"ContainerStarted","Data":"575e2349e4cf1c7c07bc74d785c8bade94d22027956713ebf950878a15305aa0"}
Nov 01 08:31:30.216836 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.216816    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8t24r" event={"ID":"608b447e-9dda-4503-bcef-26d6af784cfb","Type":"ContainerStarted","Data":"7d937ad8cda14b416ef7bdd404925f1b9e587015019b480934b0b9976104762a"}
Nov 01 08:31:30.432829 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.432728    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:29 +0000 UTC" deadline="2028-04-25 01:15:30.727086322 +0000 UTC"
Nov 01 08:31:30.432829 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.432785    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12976h44m0.294304278s"
Nov 01 08:31:30.436357 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:30.436326    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d1e97d_be9c_4f45_827c_19f789823a3b.slice/crio-85dbc996b42e9b3598d3a35a65d153a8659205a89f560396fae4db22e066868e WatchSource:0}: Error finding container 85dbc996b42e9b3598d3a35a65d153a8659205a89f560396fae4db22e066868e: Status 404 returned error can't find the container with id 85dbc996b42e9b3598d3a35a65d153a8659205a89f560396fae4db22e066868e
Nov 01 08:31:30.439317 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:30.439293    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:31:31.062492 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.062458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:31.062689 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.062516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:31.062758 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.062692    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:31.062818 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.062775    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:31:33.062747148 +0000 UTC m=+5.330150657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:31.062899 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.062856    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:31.062956 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.062909    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret podName:2e9c65da-47f6-494b-819e-34ee9896ff41 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:33.062896433 +0000 UTC m=+5.330299928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret") pod "global-pull-secret-syncer-v7gkj" (UID: "2e9c65da-47f6-494b-819e-34ee9896ff41") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:31.163959 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.163639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:31.163959 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.163814    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:31.163959 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.163832    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:31.163959 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.163846    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q6qpf for pod openshift-network-diagnostics/network-check-target-mnsws: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:31.163959 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.163929    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf podName:44a34856-dd40-4408-9db2-67d075d9850a nodeName:}" failed. No retries permitted until 2026-11-01 08:31:33.163908579 +0000 UTC m=+5.431312074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6qpf" (UniqueName: "kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf") pod "network-check-target-mnsws" (UID: "44a34856-dd40-4408-9db2-67d075d9850a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:31.203525 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.203479    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:31.203736 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.203635    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:31.203736 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.203731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:31.203861 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.203807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:31.203919 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.203867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:31.203978 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:31.203935    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:31.240691 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.240648    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vlpgj" event={"ID":"e5d1e97d-be9c-4f45-827c-19f789823a3b","Type":"ContainerStarted","Data":"85dbc996b42e9b3598d3a35a65d153a8659205a89f560396fae4db22e066868e"}
Nov 01 08:31:31.433777 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.432967    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:29 +0000 UTC" deadline="2028-05-19 08:15:23.881127849 +0000 UTC"
Nov 01 08:31:31.433777 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:31.433051    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13559h43m52.448081176s"
Nov 01 08:31:33.079351 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:33.078886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:33.079351 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:33.078944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:33.079351 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.079142    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:33.079351 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.079208    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:31:37.079189456 +0000 UTC m=+9.346592952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:33.079351 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.079274    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:33.079351 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.079313    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret podName:2e9c65da-47f6-494b-819e-34ee9896ff41 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:37.079299375 +0000 UTC m=+9.346702866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret") pod "global-pull-secret-syncer-v7gkj" (UID: "2e9c65da-47f6-494b-819e-34ee9896ff41") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:33.179644 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:33.179331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:33.179644 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.179502    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:33.179644 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.179524    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:33.179644 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.179538    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q6qpf for pod openshift-network-diagnostics/network-check-target-mnsws: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:33.179644 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.179600    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf podName:44a34856-dd40-4408-9db2-67d075d9850a nodeName:}" failed. No retries permitted until 2026-11-01 08:31:37.179581951 +0000 UTC m=+9.446985447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6qpf" (UniqueName: "kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf") pod "network-check-target-mnsws" (UID: "44a34856-dd40-4408-9db2-67d075d9850a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:33.203304 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:33.202845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:33.203304 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.202998    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:33.203304 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:33.203081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:33.203304 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.203157    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:33.203304 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:33.203199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:33.203304 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:33.203261    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:35.210989 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:35.206302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:35.210989 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:35.206681    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:35.210989 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:35.207177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:35.210989 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:35.207316    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:35.210989 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:35.207387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:35.210989 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:35.207534    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:35.276734 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:35.276698    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="fa3a14ea5192eb910d78da7a942eabe95c683f499048911036385ca5a19651ba" exitCode=0
Nov 01 08:31:35.276948 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:35.276780    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerDied","Data":"fa3a14ea5192eb910d78da7a942eabe95c683f499048911036385ca5a19651ba"}
Nov 01 08:31:35.282709 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:35.282550    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca0fe841d453cc229ff1f7a2450a3ba4" containerID="22ff066c1e5e1aa34801ff452c3132f8487894816f1ecf19854f9943f6686135" exitCode=0
Nov 01 08:31:35.282709 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:35.282651    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"ca0fe841d453cc229ff1f7a2450a3ba4","Type":"ContainerDied","Data":"22ff066c1e5e1aa34801ff452c3132f8487894816f1ecf19854f9943f6686135"}
Nov 01 08:31:35.292851 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:35.292412    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"36a0826bc10ea7285dbce782929bcb3b","Type":"ContainerStarted","Data":"a24937f6959662b79a550a9283edff0ad3011c0556d0bc5d96c5a9d18a63c24e"}
Nov 01 08:31:36.299614 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:36.299533    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"ca0fe841d453cc229ff1f7a2450a3ba4","Type":"ContainerStarted","Data":"6f74300c60dfb761454081416c6d00a9f30268ea873ea54ca20543daeb13e567"}
Nov 01 08:31:36.303140 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:36.302606    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jzchj" event={"ID":"e38ced3e-d98c-4df1-935f-a1cdc2abf313","Type":"ContainerStarted","Data":"88b36443424677d3e7b6fae7c9369d2b1e903f69d58f7266008b54d6d0b527c8"}
Nov 01 08:31:36.305932 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:36.305731    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vlpgj" event={"ID":"e5d1e97d-be9c-4f45-827c-19f789823a3b","Type":"ContainerStarted","Data":"d14f58cea4f11fb4b553aae41381a04d5cf8b8beed6589559f1fb8aa9591484f"}
Nov 01 08:31:36.416735 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:36.415637    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal" podStartSLOduration=8.415616078 podStartE2EDuration="8.415616078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:35.791003315 +0000 UTC m=+8.058406828" watchObservedRunningTime="2026-11-01 08:31:36.415616078 +0000 UTC m=+8.683019591"
Nov 01 08:31:36.535879 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:36.535004    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jzchj" podStartSLOduration=2.371675389 podStartE2EDuration="8.534985603s" totalImagesPullingTime="6.163310214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:36.534629825 +0000 UTC m=+8.802033338" watchObservedRunningTime="2026-11-01 08:31:36.534985603 +0000 UTC m=+8.802389116"
Nov 01 08:31:36.535879 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:36.535124    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal" podStartSLOduration=8.535118169 podStartE2EDuration="8.535118169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:36.416567556 +0000 UTC m=+8.683971070" watchObservedRunningTime="2026-11-01 08:31:36.535118169 +0000 UTC m=+8.802521684"
Nov 01 08:31:37.116448 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:37.116343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:37.116448 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:37.116393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:37.116671 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.116545    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:37.116671 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.116605    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:31:45.116585793 +0000 UTC m=+17.383989301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:37.117086 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.116974    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:37.117086 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.117015    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret podName:2e9c65da-47f6-494b-819e-34ee9896ff41 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:45.117004746 +0000 UTC m=+17.384408236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret") pod "global-pull-secret-syncer-v7gkj" (UID: "2e9c65da-47f6-494b-819e-34ee9896ff41") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:37.203495 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:37.203458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:37.203688 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.203610    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:37.203688 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:37.203670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:37.203762 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.203748    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:37.203841 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:37.203795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:37.203879 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.203859    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:37.216947 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:37.216922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:37.217184 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.217082    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:37.217184 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.217097    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:37.217184 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.217106    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q6qpf for pod openshift-network-diagnostics/network-check-target-mnsws: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:37.217184 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:37.217153    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf podName:44a34856-dd40-4408-9db2-67d075d9850a nodeName:}" failed. No retries permitted until 2026-11-01 08:31:45.217137548 +0000 UTC m=+17.484541052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6qpf" (UniqueName: "kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf") pod "network-check-target-mnsws" (UID: "44a34856-dd40-4408-9db2-67d075d9850a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:37.317686 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:37.317652    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh" event={"ID":"67b4458b-d683-447a-a253-5b37d8531843","Type":"ContainerStarted","Data":"8a90f0c5ea0f97a863288b3299934a5369c4fd8253684e95d30fbc7317de9fc0"}
Nov 01 08:31:39.041136 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.041108    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 08:31:39.203177 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.202735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:39.203177 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:39.202880    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:39.203177 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.202970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:39.203177 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:39.203045    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:39.203177 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.203093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:39.203177 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:39.203151    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:39.212884 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.212768    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T08:31:39.041130681Z","UUID":"eb4ae83d-0cdd-43fa-910d-ea9b9d9c8752","Handler":null,"Name":"","Endpoint":""}
Nov 01 08:31:39.214728 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.214705    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 08:31:39.214728 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.214734    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 08:31:39.324587 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.324511    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2gdwv" event={"ID":"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900","Type":"ContainerStarted","Data":"6b7bbe48ce2c7ba573cb70d45ba91fedb83966d0687135ca63b3a7872baf08f3"}
Nov 01 08:31:39.326133 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.326100    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8t24r" event={"ID":"608b447e-9dda-4503-bcef-26d6af784cfb","Type":"ContainerStarted","Data":"d892f14b173d296335c6d9ef2f4e3442958b3595fe965efbb3996ec6214b8fb7"}
Nov 01 08:31:39.328281 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.328206    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh" event={"ID":"67b4458b-d683-447a-a253-5b37d8531843","Type":"ContainerStarted","Data":"137314b3c3c9f8aaabfd5f7c2a75482366c40420bf06c7fb270fb6015002da78"}
Nov 01 08:31:39.449151 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.449103    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vlpgj" podStartSLOduration=6.528399106 podStartE2EDuration="11.449090893s" totalImagesPullingTime="4.920691787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:36.650823891 +0000 UTC m=+8.918227402" watchObservedRunningTime="2026-11-01 08:31:39.449090893 +0000 UTC m=+11.716494402"
Nov 01 08:31:39.449529 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.449500    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2gdwv" podStartSLOduration=2.603346061 podStartE2EDuration="11.449492486s" totalImagesPullingTime="8.846146425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:39.448691872 +0000 UTC m=+11.716095388" watchObservedRunningTime="2026-11-01 08:31:39.449492486 +0000 UTC m=+11.716895998"
Nov 01 08:31:39.563369 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:39.563308    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8t24r" podStartSLOduration=2.652305033 podStartE2EDuration="11.563289126s" totalImagesPullingTime="8.910984093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:39.562916712 +0000 UTC m=+11.830320228" watchObservedRunningTime="2026-11-01 08:31:39.563289126 +0000 UTC m=+11.830692639"
Nov 01 08:31:40.333464 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:40.333422    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="e62fce31f069de95945e62886490d4b998204428007322b02685b86241415e55" exitCode=0
Nov 01 08:31:40.334916 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:40.334460    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerDied","Data":"e62fce31f069de95945e62886490d4b998204428007322b02685b86241415e55"}
Nov 01 08:31:40.876226 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:40.876192    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:40.876838 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:40.876814    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:41.203886 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.202865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:41.203886 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.203071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:41.203886 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:41.203173    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:41.203886 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:41.203257    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:41.203886 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.203313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:41.203886 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:41.203411    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:41.338990 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.338950    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh" event={"ID":"67b4458b-d683-447a-a253-5b37d8531843","Type":"ContainerStarted","Data":"7e48e9af1ebfefa5ab01ad9c62f2f759fd892779410e587a2b5f26e4cc7463a3"}
Nov 01 08:31:41.341173 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.341087    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ln7w5" event={"ID":"89776b0e-b9bc-415b-8138-9ce3e9264a27","Type":"ContainerStarted","Data":"29175a50e457e6be4ab2a24474e997cb081e93e869da31b64b1b5c3c36752efd"}
Nov 01 08:31:41.342927 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.342881    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cskq4" event={"ID":"a108cdc6-7b79-4e7a-8b7d-7824406258e3","Type":"ContainerStarted","Data":"db6cc83594d2f0e06db32880b078dcb38c3eb7502f350be95f621b4bf4fb1c86"}
Nov 01 08:31:41.458002 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.457892    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh" podStartSLOduration=3.212153745 podStartE2EDuration="13.457870837s" totalImagesPullingTime="10.245717092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:41.457519035 +0000 UTC m=+13.724922559" watchObservedRunningTime="2026-11-01 08:31:41.457870837 +0000 UTC m=+13.725274396"
Nov 01 08:31:41.690504 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.690452    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ln7w5" podStartSLOduration=4.54209631 podStartE2EDuration="13.690438009s" totalImagesPullingTime="9.148341699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:41.572650679 +0000 UTC m=+13.840054191" watchObservedRunningTime="2026-11-01 08:31:41.690438009 +0000 UTC m=+13.957841521"
Nov 01 08:31:41.690665 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:41.690548    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cskq4" podStartSLOduration=2.270188486 podStartE2EDuration="13.690543295s" totalImagesPullingTime="11.420354809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:41.690278133 +0000 UTC m=+13.957681646" watchObservedRunningTime="2026-11-01 08:31:41.690543295 +0000 UTC m=+13.957946810"
Nov 01 08:31:42.347759 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:42.347719    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="86aba9dbf1dea8b53a55f25654b12d9ae0b832b31ecdff497154df7dbfeac3c9" exitCode=0
Nov 01 08:31:42.348351 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:42.347861    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerDied","Data":"86aba9dbf1dea8b53a55f25654b12d9ae0b832b31ecdff497154df7dbfeac3c9"}
Nov 01 08:31:43.203166 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:43.203134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:43.203334 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:43.203180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:43.203334 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:43.203296    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:43.203436 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:43.203350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:43.203465 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:43.203427    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:43.203525 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:43.203509    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:44.355013 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:44.354980    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="2d05e1b1fb8d7e23b7d72cb03c7aeacb0257df61f11806680512ae8e0a0b93f1" exitCode=0
Nov 01 08:31:44.355702 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:44.355066    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerDied","Data":"2d05e1b1fb8d7e23b7d72cb03c7aeacb0257df61f11806680512ae8e0a0b93f1"}
Nov 01 08:31:45.181851 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:45.181651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:45.182017 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.181795    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:45.182017 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.181970    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:32:01.181955387 +0000 UTC m=+33.449358882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 08:31:45.182017 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.181986    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:45.182017 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:45.181912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:45.182215 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.182056    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret podName:2e9c65da-47f6-494b-819e-34ee9896ff41 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:01.182019623 +0000 UTC m=+33.449423113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret") pod "global-pull-secret-syncer-v7gkj" (UID: "2e9c65da-47f6-494b-819e-34ee9896ff41") : object "kube-system"/"original-pull-secret" not registered
Nov 01 08:31:45.202805 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:45.202780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:45.202903 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:45.202862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:45.202991 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:45.202977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:45.203067 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.202989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:45.203118 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.203069    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:45.203153 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.203121    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:45.282524 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:45.282490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:45.282681 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.282634    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 08:31:45.282681 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.282652    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 08:31:45.282681 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.282664    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q6qpf for pod openshift-network-diagnostics/network-check-target-mnsws: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:45.282837 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:45.282725    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf podName:44a34856-dd40-4408-9db2-67d075d9850a nodeName:}" failed. No retries permitted until 2026-11-01 08:32:01.282706716 +0000 UTC m=+33.550110227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6qpf" (UniqueName: "kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf") pod "network-check-target-mnsws" (UID: "44a34856-dd40-4408-9db2-67d075d9850a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 08:31:46.362704 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.362544    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:31:46.363169 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.363018    2579 generic.go:356] "Generic (PLEG): container finished" podID="f13e4499-bb2f-4e37-bd37-52f4bdad02dd" containerID="82439b0904e4b96abfab98c648dc1bdf2c255be7a6e484c63c605c5ae43cad83" exitCode=1
Nov 01 08:31:46.363169 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.363065    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"2c21147ba987a2a99f3e931fff32029cab1c3c2a89378c19840b47a1bcbb086f"}
Nov 01 08:31:46.363169 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.363163    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"1fad641044cf3597219be1aee44db42c4ae3121c742d377912827605038a9d33"}
Nov 01 08:31:46.363281 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.363217    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"71e0bcff13db11ef250f9f9140e3487280b01d192eeedf9181208851b0b0b05e"}
Nov 01 08:31:46.363281 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.363267    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"f5a3149d64459daff9d990280fb0c05643ec5aef87141c176efe734e6278f1d0"}
Nov 01 08:31:46.363358 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.363313    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerDied","Data":"82439b0904e4b96abfab98c648dc1bdf2c255be7a6e484c63c605c5ae43cad83"}
Nov 01 08:31:46.363397 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.363367    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"c42c27768f67d7b791e9ff76a9b417fb08871789e0ffba52183b2f7f273c92ea"}
Nov 01 08:31:46.365940 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:46.365919    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerStarted","Data":"d7f66ef7609d20cd261c7568e05fa2bdd3a904d1cd15891b14cd03866d29c4d9"}
Nov 01 08:31:47.202983 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:47.202949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:47.203202 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:47.203075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:47.203202 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:47.203091    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:47.203202 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:47.203155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:47.203202 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:47.203161    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:47.203410 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:47.203228    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:47.370684 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:47.370642    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="d7f66ef7609d20cd261c7568e05fa2bdd3a904d1cd15891b14cd03866d29c4d9" exitCode=0
Nov 01 08:31:47.371121 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:47.370694    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerDied","Data":"d7f66ef7609d20cd261c7568e05fa2bdd3a904d1cd15891b14cd03866d29c4d9"}
Nov 01 08:31:48.375713 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:48.375677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:31:48.376304 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:48.376049    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"5c43f0f6d9f928ff7fb576cfb194b35b76d77366d00e5c0f1e0c9b9189d5e838"}
Nov 01 08:31:48.378912 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:48.378889    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="818f81de48788899a0f031371bc3222d185469b68f1c169d83b7bea1271c9779" exitCode=0
Nov 01 08:31:48.379042 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:48.378940    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerDied","Data":"818f81de48788899a0f031371bc3222d185469b68f1c169d83b7bea1271c9779"}
Nov 01 08:31:49.203274 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:49.203244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:49.203442 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:49.203279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:49.203442 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:49.203337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:49.203442 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:49.203350    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:49.203442 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:49.203397    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:49.203563 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:49.203484    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:49.384305 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:49.384267    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" event={"ID":"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e","Type":"ContainerStarted","Data":"f0f452ce33e80c3bbb4420dec4227bb6ce82847df6a29d6bc6711c24a349d769"}
Nov 01 08:31:49.557806 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:49.557712    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jnc4r" podStartSLOduration=9.108102536 podStartE2EDuration="21.557700629s" totalImagesPullingTime="12.449598093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:49.556776155 +0000 UTC m=+21.824179667" watchObservedRunningTime="2026-11-01 08:31:49.557700629 +0000 UTC m=+21.825104140"
Nov 01 08:31:50.389776 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:50.389748    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:31:50.390414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:50.390216    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"9fe3f4c01b6a6e7f287171bde5fa832600f6c97e0fe9e873894e4b3a69ef6ad7"}
Nov 01 08:31:50.390950 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:50.390783    2579 scope.go:122] "RemoveContainer" containerID="82439b0904e4b96abfab98c648dc1bdf2c255be7a6e484c63c605c5ae43cad83"
Nov 01 08:31:51.202806 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.202766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:51.203218 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.202842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:51.203218 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.202970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:51.203218 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:51.202988    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:51.203218 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:51.203096    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:51.203218 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:51.203177    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:51.395916 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.395892    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:31:51.396351 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.396268    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" event={"ID":"f13e4499-bb2f-4e37-bd37-52f4bdad02dd","Type":"ContainerStarted","Data":"fa1f103ae36683a6b9682d57e87e2502690be7b1cf5eeae5ae60974078b9571d"}
Nov 01 08:31:51.396453 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.396436    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:51.396527 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.396515    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:51.396593 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.396568    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:51.411814 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.411787    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:51.412016 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.412000    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:31:51.606960 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:51.606918    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg" podStartSLOduration=8.125154218 podStartE2EDuration="23.606905627s" totalImagesPullingTime="15.481751409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:51.606651221 +0000 UTC m=+23.874054742" watchObservedRunningTime="2026-11-01 08:31:51.606905627 +0000 UTC m=+23.874309184"
Nov 01 08:31:52.575970 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:52.575888    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mnsws"]
Nov 01 08:31:52.576592 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:52.576045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:52.576592 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:52.576159    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:52.578630 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:52.578610    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7gkj"]
Nov 01 08:31:52.578709 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:52.578699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:52.578790 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:52.578775    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:52.579119 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:52.579099    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nlwtz"]
Nov 01 08:31:52.579204 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:52.579192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:52.579288 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:52.579270    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:54.202703 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:54.202669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:54.202703 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:54.202703    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:54.203227 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:54.202668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:54.203227 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:54.202794    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:54.203227 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:54.202876    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:54.203227 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:54.202976    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:54.947266 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:54.947174    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:54.947436 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:54.947328    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 08:31:54.947873 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:54.947855    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:31:56.202685 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:56.202656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:56.203080 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:56.202726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:56.203080 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:56.202761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:56.203080 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:56.202879    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:56.203080 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:56.202922    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:56.203080 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:56.202994    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:58.204124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.204090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:31:58.204539 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.204175    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7gkj" podUID="2e9c65da-47f6-494b-819e-34ee9896ff41"
Nov 01 08:31:58.204539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.204257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:31:58.204539 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.204306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:31:58.204539 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.204393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlwtz" podUID="afb159a0-e7ce-43e5-be5f-5260297660cb"
Nov 01 08:31:58.204539 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.204437    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mnsws" podUID="44a34856-dd40-4408-9db2-67d075d9850a"
Nov 01 08:31:58.522474 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.522300    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeReady"
Nov 01 08:31:58.522637 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.522558    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 08:31:58.681042 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.680996    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fg7b2"]
Nov 01 08:31:58.683790 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.683776    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ftpbm"]
Nov 01 08:31:58.683911 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.683892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fg7b2"
Nov 01 08:31:58.686195 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.686180    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-67r69"]
Nov 01 08:31:58.686304 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.686288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ftpbm"
Nov 01 08:31:58.688468 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.688440    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-62p6x"]
Nov 01 08:31:58.688560 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.688533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-67r69"
Nov 01 08:31:58.690811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.690795    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jd869"]
Nov 01 08:31:58.692116 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.690875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.693799 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.693782    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"]
Nov 01 08:31:58.693898 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.693884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.696623 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.696609    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"]
Nov 01 08:31:58.696744 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.696727    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.699145 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.699129    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bbbls"]
Nov 01 08:31:58.699250 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.699237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.701910 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.701887    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78dd945d47-r7lx4"]
Nov 01 08:31:58.702010 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.701993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:31:58.704662 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.704645    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"]
Nov 01 08:31:58.704738 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.704725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.707909 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.707891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.731922 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.731900    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"]
Nov 01 08:31:58.734992 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.734974    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fd6466786-gpntg"]
Nov 01 08:31:58.735124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.735107    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.738203 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.738182    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"]
Nov 01 08:31:58.738466 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.738445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.740990 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.740966    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c6bf599bf-9cjkf"]
Nov 01 08:31:58.741355 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.741335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"
Nov 01 08:31:58.748640 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.746334    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.748640 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.746421    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.748640 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.746554    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 08:31:58.748640 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.746579    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.748640 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.746756    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 08:31:58.748640 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.746904    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.750044 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.750008    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"]
Nov 01 08:31:58.750271 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.750251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.753160 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.753143    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-88hrp"]
Nov 01 08:31:58.753292 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.753271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"
Nov 01 08:31:58.756198 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.756180    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8hhs6"]
Nov 01 08:31:58.756304 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.756289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:31:58.759839 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759823    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ftpbm"]
Nov 01 08:31:58.759911 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759859    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fg7b2"]
Nov 01 08:31:58.759911 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759880    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-67r69"]
Nov 01 08:31:58.759911 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759899    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-62p6x"]
Nov 01 08:31:58.760010 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759917    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"]
Nov 01 08:31:58.760010 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759944    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"]
Nov 01 08:31:58.760010 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759962    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bbbls"]
Nov 01 08:31:58.760010 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759980    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jd869"]
Nov 01 08:31:58.760010 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.760184 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.759998    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"]
Nov 01 08:31:58.780815 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.780791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-service-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.780923 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.780833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05aad278-6386-4a54-89c9-c143151e86cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:31:58.780923 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.780856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkqz\" (UniqueName: \"kubernetes.io/projected/bcd6ef98-6c6a-4687-bc85-4a540e06022a-kube-api-access-2fkqz\") pod \"kas-connection-checker-685895c7c9-ftpbm\" (UID: \"bcd6ef98-6c6a-4687-bc85-4a540e06022a\") " pod="kube-system/kas-connection-checker-685895c7c9-ftpbm"
Nov 01 08:31:58.781041 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.780996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zfzs\" (UniqueName: \"kubernetes.io/projected/f1f77890-a449-4958-9b83-ef93acbe2f58-kube-api-access-7zfzs\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.781104 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrt9t\" (UniqueName: \"kubernetes.io/projected/4f63fd1a-4af7-414d-aa39-4580e4e98d7f-kube-api-access-hrt9t\") pod \"kas-connection-checker-685895c7c9-67r69\" (UID: \"4f63fd1a-4af7-414d-aa39-4580e4e98d7f\") " pod="kube-system/kas-connection-checker-685895c7c9-67r69"
Nov 01 08:31:58.781154 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:31:58.781233 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-trusted-ca\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.781438 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.781528 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/81ea361a-8d47-4aee-8ed8-b742628152ba-kube-api-access-wvttn\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.781588 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-config\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.781588 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81ea361a-8d47-4aee-8ed8-b742628152ba-serving-cert\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.781588 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.781749 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxtr\" (UniqueName: \"kubernetes.io/projected/46b41145-f5ea-4af3-9f72-b12fee0dd5e2-kube-api-access-vgxtr\") pod \"kas-connection-checker-685895c7c9-fg7b2\" (UID: \"46b41145-f5ea-4af3-9f72-b12fee0dd5e2\") " pod="kube-system/kas-connection-checker-685895c7c9-fg7b2"
Nov 01 08:31:58.781749 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.781749 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.781749 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hdn\" (UniqueName: \"kubernetes.io/projected/de81ecfb-e731-40aa-9732-ca40772d9444-kube-api-access-67hdn\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.781934 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-tmp\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.781934 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03c05080-9769-4bf4-ac9a-39b2c7301170-serving-cert\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.781934 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526c4\" (UniqueName: \"kubernetes.io/projected/03c05080-9769-4bf4-ac9a-39b2c7301170-kube-api-access-526c4\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.781934 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1f77890-a449-4958-9b83-ef93acbe2f58-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.781934 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.781907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-snapshots\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.796157 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.796123    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 08:31:58.797528 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.797508    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fd6466786-gpntg"]
Nov 01 08:31:58.798583 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.798543    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78dd945d47-r7lx4"]
Nov 01 08:31:58.799246 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.799231    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8hhs6"]
Nov 01 08:31:58.800283 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.800265    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c6bf599bf-9cjkf"]
Nov 01 08:31:58.833781 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.833753    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"]
Nov 01 08:31:58.834393 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.834378    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"]
Nov 01 08:31:58.836527 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.836506    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"]
Nov 01 08:31:58.842736 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.842714    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.842934 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.842913    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-88hrp"]
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.842920    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.842953    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843042    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843051    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843046    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843067    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843092    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843137    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843137    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843206    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.843251 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843244    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843270    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843285    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843297    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843261    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843378    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843429    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843449    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843476    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843496    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843497    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 08:31:58.843786 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843498    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.844370 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.843984    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 08:31:58.844370 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.844054    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 08:31:58.844370 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.844235    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 08:31:58.882554 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-trusted-ca\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.882694 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae17519f-2301-43ce-ac7e-890042574c55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.882694 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.882807 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm7hr\" (UniqueName: \"kubernetes.io/projected/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-kube-api-access-xm7hr\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.882807 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qssqc\" (UniqueName: \"kubernetes.io/projected/ae17519f-2301-43ce-ac7e-890042574c55-kube-api-access-qssqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.882807 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.882921 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae17519f-2301-43ce-ac7e-890042574c55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.882921 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrt9t\" (UniqueName: \"kubernetes.io/projected/4f63fd1a-4af7-414d-aa39-4580e4e98d7f-kube-api-access-hrt9t\") pod \"kas-connection-checker-685895c7c9-67r69\" (UID: \"4f63fd1a-4af7-414d-aa39-4580e4e98d7f\") " pod="kube-system/kas-connection-checker-685895c7c9-67r69"
Nov 01 08:31:58.882988 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:31:58.883062 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.882994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.883062 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpw5p\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.883156 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.883156 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.883590 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.883692 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.883692 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.883692 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.883889 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.883889 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81ea361a-8d47-4aee-8ed8-b742628152ba-serving-cert\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.883889 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.883889 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxtr\" (UniqueName: \"kubernetes.io/projected/46b41145-f5ea-4af3-9f72-b12fee0dd5e2-kube-api-access-vgxtr\") pod \"kas-connection-checker-685895c7c9-fg7b2\" (UID: \"46b41145-f5ea-4af3-9f72-b12fee0dd5e2\") " pod="kube-system/kas-connection-checker-685895c7c9-fg7b2"
Nov 01 08:31:58.883889 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.884139 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.884139 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hdn\" (UniqueName: \"kubernetes.io/projected/de81ecfb-e731-40aa-9732-ca40772d9444-kube-api-access-67hdn\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.884139 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.883971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gmq\" (UniqueName: \"kubernetes.io/projected/8800f430-6279-4e64-aef8-7c75e914b72b-kube-api-access-69gmq\") pod \"network-check-source-6f78d59f99-twnmm\" (UID: \"8800f430-6279-4e64-aef8-7c75e914b72b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"
Nov 01 08:31:58.884139 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.884139 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03c05080-9769-4bf4-ac9a-39b2c7301170-serving-cert\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.884139 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526c4\" (UniqueName: \"kubernetes.io/projected/03c05080-9769-4bf4-ac9a-39b2c7301170-kube-api-access-526c4\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.884139 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-snapshots\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.884443 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-default-certificate\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.884443 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.884443 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.884443 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05aad278-6386-4a54-89c9-c143151e86cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:31:58.884443 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkqz\" (UniqueName: \"kubernetes.io/projected/bcd6ef98-6c6a-4687-bc85-4a540e06022a-kube-api-access-2fkqz\") pod \"kas-connection-checker-685895c7c9-ftpbm\" (UID: \"bcd6ef98-6c6a-4687-bc85-4a540e06022a\") " pod="kube-system/kas-connection-checker-685895c7c9-ftpbm"
Nov 01 08:31:58.884443 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-cert\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:31:58.884443 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.884443 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2d60e-838e-42df-8e42-58f036371709-tmp-dir\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zfzs\" (UniqueName: \"kubernetes.io/projected/f1f77890-a449-4958-9b83-ef93acbe2f58-kube-api-access-7zfzs\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-522hk\" (UniqueName: \"kubernetes.io/projected/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-kube-api-access-522hk\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-serving-cert\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/81ea361a-8d47-4aee-8ed8-b742628152ba-kube-api-access-wvttn\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-stats-auth\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpkb\" (UniqueName: \"kubernetes.io/projected/6f5bef45-538f-465e-85cf-84a97fc9ba78-kube-api-access-ctpkb\") pod \"volume-data-source-validator-58c9f5f46d-75ht8\" (UID: \"6f5bef45-538f-465e-85cf-84a97fc9ba78\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-config\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwtt\" (UniqueName: \"kubernetes.io/projected/c6f2d60e-838e-42df-8e42-58f036371709-kube-api-access-wzwtt\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.884809 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f2d60e-838e-42df-8e42-58f036371709-config-volume\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-tmp\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.884918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.885003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1f77890-a449-4958-9b83-ef93acbe2f58-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.885055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xkb\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.885090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.885126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgd5t\" (UniqueName: \"kubernetes.io/projected/3ca1667a-32ed-4777-8755-ec7e72add722-kube-api-access-kgd5t\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.885163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-service-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.885312 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.885204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-config\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.886536 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.886515    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.887156 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.887135    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 08:31:58.887414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.887388    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 08:31:58.887522 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.887501    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-tmp\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.887844 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.887821    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-config\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.887928 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.887851    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-service-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.888333 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.888313    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05aad278-6386-4a54-89c9-c143151e86cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:31:58.888516 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.888496    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-snapshots\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.888934 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.888917    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1f77890-a449-4958-9b83-ef93acbe2f58-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.889111 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.889093    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.890468 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.890452    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81ea361a-8d47-4aee-8ed8-b742628152ba-serving-cert\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.892146 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.892128    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 08:31:58.892215 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.892191    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 08:31:58.893107 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893070    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 08:31:58.893245 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893224    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 08:31:58.893396 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893379    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 08:31:58.893459 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.893289    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 08:31:58.893459 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893272    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 08:31:58.893459 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.893457    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert podName:05aad278-6386-4a54-89c9-c143151e86cf nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.393438153 +0000 UTC m=+31.660841680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bbbls" (UID: "05aad278-6386-4a54-89c9-c143151e86cf") : secret "networking-console-plugin-cert" not found
Nov 01 08:31:58.893651 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893625    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 08:31:58.893793 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893696    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 08:31:58.893793 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893705    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8lh4f\""
Nov 01 08:31:58.893793 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893714    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mnx4s\""
Nov 01 08:31:58.893793 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893767    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h4rfg\""
Nov 01 08:31:58.893793 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893770    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893845    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893847    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jftgd\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893869    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893879    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893900    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893904    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-glvfr\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893909    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-r8vnl\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893930    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pn77d\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893927    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-trusted-ca\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893948    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893977    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fph9z\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.893998    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ngh75\""
Nov 01 08:31:58.894165 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.894071    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-52479\""
Nov 01 08:31:58.898290 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.898275    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:31:58.898372 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.898329    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.398313506 +0000 UTC m=+31.665717004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:31:58.900217 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.900199    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03c05080-9769-4bf4-ac9a-39b2c7301170-serving-cert\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.903081 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.903065    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:31:58.903172 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.903116    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.40310536 +0000 UTC m=+31.670508849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:31:58.949850 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.949821    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkqz\" (UniqueName: \"kubernetes.io/projected/bcd6ef98-6c6a-4687-bc85-4a540e06022a-kube-api-access-2fkqz\") pod \"kas-connection-checker-685895c7c9-ftpbm\" (UID: \"bcd6ef98-6c6a-4687-bc85-4a540e06022a\") " pod="kube-system/kas-connection-checker-685895c7c9-ftpbm"
Nov 01 08:31:58.950185 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.950161    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrt9t\" (UniqueName: \"kubernetes.io/projected/4f63fd1a-4af7-414d-aa39-4580e4e98d7f-kube-api-access-hrt9t\") pod \"kas-connection-checker-685895c7c9-67r69\" (UID: \"4f63fd1a-4af7-414d-aa39-4580e4e98d7f\") " pod="kube-system/kas-connection-checker-685895c7c9-67r69"
Nov 01 08:31:58.950265 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.950198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/81ea361a-8d47-4aee-8ed8-b742628152ba-kube-api-access-wvttn\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:58.950265 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.950212    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hdn\" (UniqueName: \"kubernetes.io/projected/de81ecfb-e731-40aa-9732-ca40772d9444-kube-api-access-67hdn\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:58.950265 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.950248    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxtr\" (UniqueName: \"kubernetes.io/projected/46b41145-f5ea-4af3-9f72-b12fee0dd5e2-kube-api-access-vgxtr\") pod \"kas-connection-checker-685895c7c9-fg7b2\" (UID: \"46b41145-f5ea-4af3-9f72-b12fee0dd5e2\") " pod="kube-system/kas-connection-checker-685895c7c9-fg7b2"
Nov 01 08:31:58.950780 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.950760    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526c4\" (UniqueName: \"kubernetes.io/projected/03c05080-9769-4bf4-ac9a-39b2c7301170-kube-api-access-526c4\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:58.950978 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.950962    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zfzs\" (UniqueName: \"kubernetes.io/projected/f1f77890-a449-4958-9b83-ef93acbe2f58-kube-api-access-7zfzs\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:58.985676 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.985653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-stats-auth\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.985797 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.985682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpkb\" (UniqueName: \"kubernetes.io/projected/6f5bef45-538f-465e-85cf-84a97fc9ba78-kube-api-access-ctpkb\") pod \"volume-data-source-validator-58c9f5f46d-75ht8\" (UID: \"6f5bef45-538f-465e-85cf-84a97fc9ba78\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"
Nov 01 08:31:58.985797 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.985703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.985797 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.985720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwtt\" (UniqueName: \"kubernetes.io/projected/c6f2d60e-838e-42df-8e42-58f036371709-kube-api-access-wzwtt\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.985958 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.985805    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:31:58.985958 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.985824    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dd945d47-r7lx4: secret "image-registry-tls" not found
Nov 01 08:31:58.985958 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.985877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f2d60e-838e-42df-8e42-58f036371709-config-volume\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.985958 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.985892    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls podName:5974cc8f-4fe4-4143-bd79-d1b8888fc1ab nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.485873886 +0000 UTC m=+31.753277393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls") pod "image-registry-78dd945d47-r7lx4" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab") : secret "image-registry-tls" not found
Nov 01 08:31:58.985958 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.985930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.985962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xkb\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgd5t\" (UniqueName: \"kubernetes.io/projected/3ca1667a-32ed-4777-8755-ec7e72add722-kube-api-access-kgd5t\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-config\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae17519f-2301-43ce-ac7e-890042574c55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm7hr\" (UniqueName: \"kubernetes.io/projected/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-kube-api-access-xm7hr\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qssqc\" (UniqueName: \"kubernetes.io/projected/ae17519f-2301-43ce-ac7e-890042574c55-kube-api-access-qssqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.986218    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.486201294 +0000 UTC m=+31.753604784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae17519f-2301-43ce-ac7e-890042574c55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.986291 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986290    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpw5p\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986576    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f2d60e-838e-42df-8e42-58f036371709-config-volume\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gmq\" (UniqueName: \"kubernetes.io/projected/8800f430-6279-4e64-aef8-7c75e914b72b-kube-api-access-69gmq\") pod \"network-check-source-6f78d59f99-twnmm\" (UID: \"8800f430-6279-4e64-aef8-7c75e914b72b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986794    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae17519f-2301-43ce-ac7e-890042574c55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.986811 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-default-certificate\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-cert\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986934    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.987120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2d60e-838e-42df-8e42-58f036371709-tmp-dir\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.987161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-522hk\" (UniqueName: \"kubernetes.io/projected/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-kube-api-access-522hk\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.987185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-serving-cert\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.987206    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.986686    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-config\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.987519    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2d60e-838e-42df-8e42-58f036371709-tmp-dir\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.986889    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:31:58.987680 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.987319    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:31:58.988304 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.987721    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.988304 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.987837    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.487821966 +0000 UTC m=+31.755225474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:31:58.989354 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.988638    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-stats-auth\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.989354 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.989041    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.489003715 +0000 UTC m=+31.756407206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:31:58.989354 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.989111    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:31:58.989354 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.989124    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd6466786-gpntg: secret "image-registry-tls" not found
Nov 01 08:31:58.989354 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:58.989169    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls podName:6ddfd7c8-e096-40dc-8e3d-a359cb566176 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:59.489156516 +0000 UTC m=+31.756560008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls") pod "image-registry-6fd6466786-gpntg" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176") : secret "image-registry-tls" not found
Nov 01 08:31:58.989354 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.989174    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.989354 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.989304    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.990143 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.990099    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.990341 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.990309    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-serving-cert\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:58.990828 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.990805    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-cert\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:31:58.991046 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.991008    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.991103 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.991045    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae17519f-2301-43ce-ac7e-890042574c55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:58.991191 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.991174    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-default-certificate\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:58.991523 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.991504    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:58.991583 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.991510    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:58.998470 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:58.998454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fg7b2"
Nov 01 08:31:59.003728 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.003713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ftpbm"
Nov 01 08:31:59.009104 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.009089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-67r69"
Nov 01 08:31:59.015896 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.015878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:31:59.022216 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.021828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:31:59.059092 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.056448    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-522hk\" (UniqueName: \"kubernetes.io/projected/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-kube-api-access-522hk\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:31:59.059092 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.057039    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gmq\" (UniqueName: \"kubernetes.io/projected/8800f430-6279-4e64-aef8-7c75e914b72b-kube-api-access-69gmq\") pod \"network-check-source-6f78d59f99-twnmm\" (UID: \"8800f430-6279-4e64-aef8-7c75e914b72b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"
Nov 01 08:31:59.059092 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.057791    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwtt\" (UniqueName: \"kubernetes.io/projected/c6f2d60e-838e-42df-8e42-58f036371709-kube-api-access-wzwtt\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:59.059092 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.058343    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgd5t\" (UniqueName: \"kubernetes.io/projected/3ca1667a-32ed-4777-8755-ec7e72add722-kube-api-access-kgd5t\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:59.063381 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.063359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpw5p\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:59.069233 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.067976    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:59.069233 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.068050    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm7hr\" (UniqueName: \"kubernetes.io/projected/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-kube-api-access-xm7hr\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:59.069897 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.069645    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpkb\" (UniqueName: \"kubernetes.io/projected/6f5bef45-538f-465e-85cf-84a97fc9ba78-kube-api-access-ctpkb\") pod \"volume-data-source-validator-58c9f5f46d-75ht8\" (UID: \"6f5bef45-538f-465e-85cf-84a97fc9ba78\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"
Nov 01 08:31:59.070526 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.070479    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qssqc\" (UniqueName: \"kubernetes.io/projected/ae17519f-2301-43ce-ac7e-890042574c55-kube-api-access-qssqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:59.070735 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.070711    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xkb\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:59.072784 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.072757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:31:59.074316 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.074274    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:59.086505 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.086312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"
Nov 01 08:31:59.100688 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.100623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"
Nov 01 08:31:59.108615 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.108264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:31:59.365998 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.365958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:31:59.393741 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.393718    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-67r69"]
Nov 01 08:31:59.393968 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:59.393947    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f63fd1a_4af7_414d_aa39_4580e4e98d7f.slice/crio-097de667967bfd80c5a30fca7770d38bf52636bc413628f0db6c9fd1b1971dbd WatchSource:0}: Error finding container 097de667967bfd80c5a30fca7770d38bf52636bc413628f0db6c9fd1b1971dbd: Status 404 returned error can't find the container with id 097de667967bfd80c5a30fca7770d38bf52636bc413628f0db6c9fd1b1971dbd
Nov 01 08:31:59.414223 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.414148    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ftpbm"]
Nov 01 08:31:59.429218 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:59.428559    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd6ef98_6c6a_4687_bc85_4a540e06022a.slice/crio-d35f7f8dab151f28f44aa9a14af073299847bda28c79fe56a0c340e594ed5373 WatchSource:0}: Error finding container d35f7f8dab151f28f44aa9a14af073299847bda28c79fe56a0c340e594ed5373: Status 404 returned error can't find the container with id d35f7f8dab151f28f44aa9a14af073299847bda28c79fe56a0c340e594ed5373
Nov 01 08:31:59.430470 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.430448    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fg7b2"]
Nov 01 08:31:59.432342 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.432304    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-62p6x"]
Nov 01 08:31:59.454521 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.447937    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-67r69" event={"ID":"4f63fd1a-4af7-414d-aa39-4580e4e98d7f","Type":"ContainerStarted","Data":"097de667967bfd80c5a30fca7770d38bf52636bc413628f0db6c9fd1b1971dbd"}
Nov 01 08:31:59.454521 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.451083    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jd869"]
Nov 01 08:31:59.462330 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:59.462277    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c05080_9769_4bf4_ac9a_39b2c7301170.slice/crio-b5633325636bca4b69234a0fe51e659a0248a47f65b9df77c867137f94425011 WatchSource:0}: Error finding container b5633325636bca4b69234a0fe51e659a0248a47f65b9df77c867137f94425011: Status 404 returned error can't find the container with id b5633325636bca4b69234a0fe51e659a0248a47f65b9df77c867137f94425011
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.492396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.492470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.492504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.492539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.492576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.492627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.492650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.492675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.492814    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.492838    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd6466786-gpntg: secret "image-registry-tls" not found
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.492914    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls podName:6ddfd7c8-e096-40dc-8e3d-a359cb566176 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.492893884 +0000 UTC m=+32.760297400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls") pod "image-registry-6fd6466786-gpntg" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176") : secret "image-registry-tls" not found
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.492982    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493015    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.493004151 +0000 UTC m=+32.760407657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493094    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493104    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dd945d47-r7lx4: secret "image-registry-tls" not found
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493134    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls podName:5974cc8f-4fe4-4143-bd79-d1b8888fc1ab nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.493124187 +0000 UTC m=+32.760527694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls") pod "image-registry-78dd945d47-r7lx4" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab") : secret "image-registry-tls" not found
Nov 01 08:31:59.493414 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493190    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493221    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.493212025 +0000 UTC m=+32.760615534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493285    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.493275564 +0000 UTC m=+32.760679068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493325    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493336    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493367    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.493356582 +0000 UTC m=+32.760760087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493383    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert podName:05aad278-6386-4a54-89c9-c143151e86cf nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.493374239 +0000 UTC m=+32.760777731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bbbls" (UID: "05aad278-6386-4a54-89c9-c143151e86cf") : secret "networking-console-plugin-cert" not found
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493415    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:31:59.493460    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:00.49344338 +0000 UTC m=+32.760846888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:31:59.494245 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:59.493533    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae17519f_2301_43ce_ac7e_890042574c55.slice/crio-349ec0d7c233c56580a5860c15954b00d5f202a43428fd892362a68951f63791 WatchSource:0}: Error finding container 349ec0d7c233c56580a5860c15954b00d5f202a43428fd892362a68951f63791: Status 404 returned error can't find the container with id 349ec0d7c233c56580a5860c15954b00d5f202a43428fd892362a68951f63791
Nov 01 08:31:59.495136 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.495115    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"]
Nov 01 08:31:59.518426 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.518367    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"]
Nov 01 08:31:59.521569 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.521060    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"]
Nov 01 08:31:59.523557 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:59.523493    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5bef45_538f_465e_85cf_84a97fc9ba78.slice/crio-5b8f18bd537e454867d387c2eb85a01a2cd7ba5c5451ba7937d71dda0c597987 WatchSource:0}: Error finding container 5b8f18bd537e454867d387c2eb85a01a2cd7ba5c5451ba7937d71dda0c597987: Status 404 returned error can't find the container with id 5b8f18bd537e454867d387c2eb85a01a2cd7ba5c5451ba7937d71dda0c597987
Nov 01 08:31:59.527182 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.527158    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-88hrp"]
Nov 01 08:31:59.534444 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:59.534416    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a037fa_abd6_4bc2_9bc1_0d7da8429b09.slice/crio-1e2b5548b6a635aae3659ed3746e328f8dbd2b4f82ef17f43766d83c63a5ede4 WatchSource:0}: Error finding container 1e2b5548b6a635aae3659ed3746e328f8dbd2b4f82ef17f43766d83c63a5ede4: Status 404 returned error can't find the container with id 1e2b5548b6a635aae3659ed3746e328f8dbd2b4f82ef17f43766d83c63a5ede4
Nov 01 08:31:59.736162 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:31:59.736130    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2023ca6f_9936_4e4c_80a1_e33b47fa7dcb.slice/crio-763cdd6a30b86a98065bdd758b715a6495628f53b36ca23bacbc8f661684cbff WatchSource:0}: Error finding container 763cdd6a30b86a98065bdd758b715a6495628f53b36ca23bacbc8f661684cbff: Status 404 returned error can't find the container with id 763cdd6a30b86a98065bdd758b715a6495628f53b36ca23bacbc8f661684cbff
Nov 01 08:31:59.736735 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:31:59.736701    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"]
Nov 01 08:32:00.203090 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.203059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:00.203280 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.203074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:00.203280 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.203187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:00.256937 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.256906    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jcrrb\""
Nov 01 08:32:00.256937 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.256920    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 08:32:00.256937 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.256938    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xrtv\""
Nov 01 08:32:00.257183 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.256990    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 08:32:00.257183 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.257104    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xx5dr\""
Nov 01 08:32:00.451418 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.451386    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8" event={"ID":"6f5bef45-538f-465e-85cf-84a97fc9ba78","Type":"ContainerStarted","Data":"5b8f18bd537e454867d387c2eb85a01a2cd7ba5c5451ba7937d71dda0c597987"}
Nov 01 08:32:00.452490 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.452468    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8" event={"ID":"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb","Type":"ContainerStarted","Data":"763cdd6a30b86a98065bdd758b715a6495628f53b36ca23bacbc8f661684cbff"}
Nov 01 08:32:00.453657 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.453603    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4" event={"ID":"ae17519f-2301-43ce-ac7e-890042574c55","Type":"ContainerStarted","Data":"349ec0d7c233c56580a5860c15954b00d5f202a43428fd892362a68951f63791"}
Nov 01 08:32:00.454705 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.454685    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" event={"ID":"03c05080-9769-4bf4-ac9a-39b2c7301170","Type":"ContainerStarted","Data":"b5633325636bca4b69234a0fe51e659a0248a47f65b9df77c867137f94425011"}
Nov 01 08:32:00.456106 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.456088    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fg7b2" event={"ID":"46b41145-f5ea-4af3-9f72-b12fee0dd5e2","Type":"ContainerStarted","Data":"e21db7fe79a363cb4f428af5eaa855fef2b176b8378ff63164acd27fa7dfc398"}
Nov 01 08:32:00.456194 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.456115    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fg7b2" event={"ID":"46b41145-f5ea-4af3-9f72-b12fee0dd5e2","Type":"ContainerStarted","Data":"c061bb355e52e7c4853085bc2729e0732d97e901551b8b8281007f205f985446"}
Nov 01 08:32:00.457527 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.457509    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-67r69" event={"ID":"4f63fd1a-4af7-414d-aa39-4580e4e98d7f","Type":"ContainerStarted","Data":"8e152f8a3e225a2fd400cec88ae394a581579eb428deaa4dd504cb8641c8441e"}
Nov 01 08:32:00.458680 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.458661    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-88hrp" event={"ID":"29a037fa-abd6-4bc2-9bc1-0d7da8429b09","Type":"ContainerStarted","Data":"1e2b5548b6a635aae3659ed3746e328f8dbd2b4f82ef17f43766d83c63a5ede4"}
Nov 01 08:32:00.459881 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.459863    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm" event={"ID":"8800f430-6279-4e64-aef8-7c75e914b72b","Type":"ContainerStarted","Data":"ba34e5192d359b06a15fce69f50ebebdc276cb8717a6d1bcb2779ef95cfa5ea0"}
Nov 01 08:32:00.461373 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.461348    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ftpbm" event={"ID":"bcd6ef98-6c6a-4687-bc85-4a540e06022a","Type":"ContainerStarted","Data":"98633e819dbe73899cd62cd2e95ecc2a55667bb60ed8a12420207bebfecad101"}
Nov 01 08:32:00.461471 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.461386    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ftpbm" event={"ID":"bcd6ef98-6c6a-4687-bc85-4a540e06022a","Type":"ContainerStarted","Data":"d35f7f8dab151f28f44aa9a14af073299847bda28c79fe56a0c340e594ed5373"}
Nov 01 08:32:00.462516 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.462497    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-62p6x" event={"ID":"81ea361a-8d47-4aee-8ed8-b742628152ba","Type":"ContainerStarted","Data":"5d26fa0cc009b42bca2381546b1ad9583cc382aa66df9aaa1a530d4ef88e094d"}
Nov 01 08:32:00.501991 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.501959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:00.502136 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.502044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:00.502136 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.502087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:00.502136 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502114    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:00.502136 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502117    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:00.502136 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502135    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dd945d47-r7lx4: secret "image-registry-tls" not found
Nov 01 08:32:00.502287 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.502119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:00.502287 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502170    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.502152274 +0000 UTC m=+34.769555781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:00.502287 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502191    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls podName:5974cc8f-4fe4-4143-bd79-d1b8888fc1ab nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.502182235 +0000 UTC m=+34.769585737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls") pod "image-registry-78dd945d47-r7lx4" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab") : secret "image-registry-tls" not found
Nov 01 08:32:00.502287 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502205    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:00.502287 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502208    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.502199137 +0000 UTC m=+34.769602632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:32:00.502287 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502262    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.502250122 +0000 UTC m=+34.769653614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:00.502459 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.502305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:00.502459 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.502342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:00.502459 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.502360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:00.502459 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.502383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:00.502459 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502402    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:32:00.502459 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502412    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 08:32:00.502459 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502428    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.502421149 +0000 UTC m=+34.769824678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:32:00.502459 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502445    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert podName:05aad278-6386-4a54-89c9-c143151e86cf nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.502434275 +0000 UTC m=+34.769837792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bbbls" (UID: "05aad278-6386-4a54-89c9-c143151e86cf") : secret "networking-console-plugin-cert" not found
Nov 01 08:32:00.502669 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502475    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:00.502669 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502476    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:32:00.502669 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502483    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd6466786-gpntg: secret "image-registry-tls" not found
Nov 01 08:32:00.502669 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502506    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.502500176 +0000 UTC m=+34.769903665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:32:00.502669 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:00.502516    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls podName:6ddfd7c8-e096-40dc-8e3d-a359cb566176 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:02.502511231 +0000 UTC m=+34.769914720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls") pod "image-registry-6fd6466786-gpntg" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176") : secret "image-registry-tls" not found
Nov 01 08:32:00.822241 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:00.822130    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-67r69" podStartSLOduration=413.822110308 podStartE2EDuration="6m53.822110308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:00.820685378 +0000 UTC m=+33.088088884" watchObservedRunningTime="2026-11-01 08:32:00.822110308 +0000 UTC m=+33.089513840"
Nov 01 08:32:01.055873 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.055811    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fg7b2" podStartSLOduration=414.055790687 podStartE2EDuration="6m54.055790687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:00.938555405 +0000 UTC m=+33.205958917" watchObservedRunningTime="2026-11-01 08:32:01.055790687 +0000 UTC m=+33.323194198"
Nov 01 08:32:01.056096 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.056057    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ftpbm" podStartSLOduration=414.056046958 podStartE2EDuration="6m54.056046958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:01.054408791 +0000 UTC m=+33.321812303" watchObservedRunningTime="2026-11-01 08:32:01.056046958 +0000 UTC m=+33.323450470"
Nov 01 08:32:01.211855 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.210902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:01.211855 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.211053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:01.211855 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:01.211216    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:01.211855 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:01.211275    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:32:33.211256042 +0000 UTC m=+65.478659546 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : secret "metrics-daemon-secret" not found
Nov 01 08:32:01.226772 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.222413    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:01.312537 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.312501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:01.323899 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.317195    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:01.416427 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.416397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:01.423048 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.421196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:01.820530 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.820419    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7gkj"]
Nov 01 08:32:01.858908 ip-10-0-143-244 kubenswrapper[2579]: W1101 08:32:01.858856    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a34856_dd40_4408_9db2_67d075d9850a.slice/crio-0d999c2443cb79a08db1c59a7aacbef5988de6fa15087b5fde1bd683cea26b9e WatchSource:0}: Error finding container 0d999c2443cb79a08db1c59a7aacbef5988de6fa15087b5fde1bd683cea26b9e: Status 404 returned error can't find the container with id 0d999c2443cb79a08db1c59a7aacbef5988de6fa15087b5fde1bd683cea26b9e
Nov 01 08:32:01.859305 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:01.859179    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mnsws"]
Nov 01 08:32:02.485294 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.485218    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mnsws" event={"ID":"44a34856-dd40-4408-9db2-67d075d9850a","Type":"ContainerStarted","Data":"0d999c2443cb79a08db1c59a7aacbef5988de6fa15087b5fde1bd683cea26b9e"}
Nov 01 08:32:02.489244 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.488341    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8" event={"ID":"6f5bef45-538f-465e-85cf-84a97fc9ba78","Type":"ContainerStarted","Data":"6a91ce3c1dff2cb5af91c00926dd4c08142f5df0615b74b430c4cffe95ce09f7"}
Nov 01 08:32:02.493112 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.492041    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7gkj" event={"ID":"2e9c65da-47f6-494b-819e-34ee9896ff41","Type":"ContainerStarted","Data":"e005754778a8af730b692a521ffe6390c5d442ccb3f8b3fb4093c66edc2ad170"}
Nov 01 08:32:02.525308 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.525237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:02.525481 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.525329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:02.525481 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.525370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:02.525481 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.525406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:02.525481 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.525448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:02.525481 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.525473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:02.525735 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.525500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:02.525735 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.525565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:02.525735 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.525708    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:02.525875 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.525766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.525749248 +0000 UTC m=+38.793152741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:02.526246 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526225    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:02.526246 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526247    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dd945d47-r7lx4: secret "image-registry-tls" not found
Nov 01 08:32:02.526402 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526294    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls podName:5974cc8f-4fe4-4143-bd79-d1b8888fc1ab nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.526280234 +0000 UTC m=+38.793683746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls") pod "image-registry-78dd945d47-r7lx4" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab") : secret "image-registry-tls" not found
Nov 01 08:32:02.526529 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526419    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:02.526529 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.526445724 +0000 UTC m=+38.793849219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:02.526529 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526520    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.526510154 +0000 UTC m=+38.793913647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:32:02.526756 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526583    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:32:02.526756 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.526602869 +0000 UTC m=+38.794006359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:32:02.526756 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526687    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 08:32:02.526756 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526718    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert podName:05aad278-6386-4a54-89c9-c143151e86cf nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.526708745 +0000 UTC m=+38.794112240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bbbls" (UID: "05aad278-6386-4a54-89c9-c143151e86cf") : secret "networking-console-plugin-cert" not found
Nov 01 08:32:02.526958 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526768    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:32:02.526958 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526796    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.526786858 +0000 UTC m=+38.794190353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:32:02.526958 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526853    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:02.526958 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526863    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd6466786-gpntg: secret "image-registry-tls" not found
Nov 01 08:32:02.526958 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:02.526889    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls podName:6ddfd7c8-e096-40dc-8e3d-a359cb566176 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:06.52688087 +0000 UTC m=+38.794284360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls") pod "image-registry-6fd6466786-gpntg" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176") : secret "image-registry-tls" not found
Nov 01 08:32:02.605533 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:02.605466    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8" podStartSLOduration=383.096806575 podStartE2EDuration="6m25.605447907s" totalImagesPullingTime="2.508641332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:02.605257875 +0000 UTC m=+34.872661388" watchObservedRunningTime="2026-11-01 08:32:02.605447907 +0000 UTC m=+34.872851422"
Nov 01 08:32:05.506898 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:05.506860    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-62p6x" event={"ID":"81ea361a-8d47-4aee-8ed8-b742628152ba","Type":"ContainerStarted","Data":"88ebb616a07f53bab13aca4dc7a14dceab4812da00e9382ae0689c3898125851"}
Nov 01 08:32:05.511397 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:05.511173    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4" event={"ID":"ae17519f-2301-43ce-ac7e-890042574c55","Type":"ContainerStarted","Data":"60eef37798274a5c29e3dcb1c94118de1ca965856ed1b1ae63130558e0d29ad6"}
Nov 01 08:32:05.514411 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:05.514384    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/0.log"
Nov 01 08:32:05.514518 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:05.514422    2579 generic.go:356] "Generic (PLEG): container finished" podID="03c05080-9769-4bf4-ac9a-39b2c7301170" containerID="281574ad0895a2bdccb24d9c7d321c5fccaab2c09b5806fede36d302c30d4bcb" exitCode=255
Nov 01 08:32:05.514518 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:05.514465    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" event={"ID":"03c05080-9769-4bf4-ac9a-39b2c7301170","Type":"ContainerDied","Data":"281574ad0895a2bdccb24d9c7d321c5fccaab2c09b5806fede36d302c30d4bcb"}
Nov 01 08:32:05.514766 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:05.514750    2579 scope.go:122] "RemoveContainer" containerID="281574ad0895a2bdccb24d9c7d321c5fccaab2c09b5806fede36d302c30d4bcb"
Nov 01 08:32:05.642637 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:05.641221    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-62p6x" podStartSLOduration=378.377228582 podStartE2EDuration="6m23.641201209s" totalImagesPullingTime="5.263972627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:05.639773343 +0000 UTC m=+37.907176855" watchObservedRunningTime="2026-11-01 08:32:05.641201209 +0000 UTC m=+37.908604722"
Nov 01 08:32:05.882791 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:05.881490    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4" podStartSLOduration=374.348573577 podStartE2EDuration="6m19.881468609s" totalImagesPullingTime="5.532895032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:05.880411355 +0000 UTC m=+38.147814892" watchObservedRunningTime="2026-11-01 08:32:05.881468609 +0000 UTC m=+38.148872126"
Nov 01 08:32:06.356313 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.356275    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4g77m"]
Nov 01 08:32:06.388789 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.388757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.415864 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.415833    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4g77m"]
Nov 01 08:32:06.448513 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.448471    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 08:32:06.464284 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.464254    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 08:32:06.464865 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.464826    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kmhw9\""
Nov 01 08:32:06.492541 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.492508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08abdd05-c01c-4c69-ac14-330e24384e83-data-volume\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.492710 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.492676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.492767 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.492729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08abdd05-c01c-4c69-ac14-330e24384e83-crio-socket\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.492905 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.492887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmds\" (UniqueName: \"kubernetes.io/projected/08abdd05-c01c-4c69-ac14-330e24384e83-kube-api-access-ndmds\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.492969 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.492923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.520232 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.520177    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8" event={"ID":"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb","Type":"ContainerStarted","Data":"6a135f829f5c44cf468f94bb5bb7c22782ae90a46d57dfc7dfcab0c6eb465a38"}
Nov 01 08:32:06.522661 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.522635    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/1.log"
Nov 01 08:32:06.523151 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.523134    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/0.log"
Nov 01 08:32:06.523226 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.523170    2579 generic.go:356] "Generic (PLEG): container finished" podID="03c05080-9769-4bf4-ac9a-39b2c7301170" containerID="92603f6a7e867a98d077b9a5c80c979a19f2d98a80e4b0aff735d8291e6bd7e0" exitCode=255
Nov 01 08:32:06.523468 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.523443    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" event={"ID":"03c05080-9769-4bf4-ac9a-39b2c7301170","Type":"ContainerDied","Data":"92603f6a7e867a98d077b9a5c80c979a19f2d98a80e4b0aff735d8291e6bd7e0"}
Nov 01 08:32:06.523530 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.523512    2579 scope.go:122] "RemoveContainer" containerID="281574ad0895a2bdccb24d9c7d321c5fccaab2c09b5806fede36d302c30d4bcb"
Nov 01 08:32:06.523585 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.523549    2579 scope.go:122] "RemoveContainer" containerID="92603f6a7e867a98d077b9a5c80c979a19f2d98a80e4b0aff735d8291e6bd7e0"
Nov 01 08:32:06.523781 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.523750    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jd869_openshift-console-operator(03c05080-9769-4bf4-ac9a-39b2c7301170)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" podUID="03c05080-9769-4bf4-ac9a-39b2c7301170"
Nov 01 08:32:06.525883 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.525860    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-88hrp" event={"ID":"29a037fa-abd6-4bc2-9bc1-0d7da8429b09","Type":"ContainerStarted","Data":"2c41c6f5682539fa2ec889158b99241ec14a5a75a494e37d85d9f392b0beeef8"}
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.593845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.593936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.593985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08abdd05-c01c-4c69-ac14-330e24384e83-crio-socket\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594256    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594311    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:07.094290776 +0000 UTC m=+39.361694281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594369    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594382    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dd945d47-r7lx4: secret "image-registry-tls" not found
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594423    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls podName:5974cc8f-4fe4-4143-bd79-d1b8888fc1ab nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.594409498 +0000 UTC m=+46.861812987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls") pod "image-registry-78dd945d47-r7lx4" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab") : secret "image-registry-tls" not found
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmds\" (UniqueName: \"kubernetes.io/projected/08abdd05-c01c-4c69-ac14-330e24384e83-kube-api-access-ndmds\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.595378 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594584    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08abdd05-c01c-4c69-ac14-330e24384e83-crio-socket\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594647    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.594631756 +0000 UTC m=+46.862035272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594662    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594686    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594694    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594708    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:07.094701304 +0000 UTC m=+39.362104807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : configmap "kube-rbac-proxy" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594731    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.5947224 +0000 UTC m=+46.862125893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594748    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.594742438 +0000 UTC m=+46.862145927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08abdd05-c01c-4c69-ac14-330e24384e83-data-volume\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.594822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594900    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594925    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594960    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594976    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd6466786-gpntg: secret "image-registry-tls" not found
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.595011    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08abdd05-c01c-4c69-ac14-330e24384e83-data-volume\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.596001 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.594929    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.594920128 +0000 UTC m=+46.862323649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:06.596609 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.595115    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls podName:6ddfd7c8-e096-40dc-8e3d-a359cb566176 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.595097127 +0000 UTC m=+46.862500625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls") pod "image-registry-6fd6466786-gpntg" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176") : secret "image-registry-tls" not found
Nov 01 08:32:06.596609 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.595138    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.595122445 +0000 UTC m=+46.862525941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:32:06.596609 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.595296    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 08:32:06.596609 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:06.595338    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert podName:05aad278-6386-4a54-89c9-c143151e86cf nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.595325235 +0000 UTC m=+46.862728758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bbbls" (UID: "05aad278-6386-4a54-89c9-c143151e86cf") : secret "networking-console-plugin-cert" not found
Nov 01 08:32:06.637318 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.637213    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8" podStartSLOduration=374.967582943 podStartE2EDuration="6m20.637192573s" totalImagesPullingTime="5.66960963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:06.636373711 +0000 UTC m=+38.903777242" watchObservedRunningTime="2026-11-01 08:32:06.637192573 +0000 UTC m=+38.904596085"
Nov 01 08:32:06.660102 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.660066    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmds\" (UniqueName: \"kubernetes.io/projected/08abdd05-c01c-4c69-ac14-330e24384e83-kube-api-access-ndmds\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:06.708979 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.708948    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"]
Nov 01 08:32:06.726066 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.726020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"
Nov 01 08:32:06.766608 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.766565    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"]
Nov 01 08:32:06.789901 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.789853    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:06.789901 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.789890    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8rfsj\""
Nov 01 08:32:06.790124 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.789927    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 08:32:06.796728 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.796699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnngc\" (UniqueName: \"kubernetes.io/projected/2236c0f5-aab5-4979-8131-97477b58a186-kube-api-access-qnngc\") pod \"migrator-66df5d496b-sv7mk\" (UID: \"2236c0f5-aab5-4979-8131-97477b58a186\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"
Nov 01 08:32:06.873190 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.873140    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-88hrp" podStartSLOduration=2.764391687 podStartE2EDuration="8.873123018s" totalImagesPullingTime="6.108731331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:06.872198385 +0000 UTC m=+39.139601899" watchObservedRunningTime="2026-11-01 08:32:06.873123018 +0000 UTC m=+39.140526531"
Nov 01 08:32:06.898062 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.898012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnngc\" (UniqueName: \"kubernetes.io/projected/2236c0f5-aab5-4979-8131-97477b58a186-kube-api-access-qnngc\") pod \"migrator-66df5d496b-sv7mk\" (UID: \"2236c0f5-aab5-4979-8131-97477b58a186\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"
Nov 01 08:32:06.959464 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:06.959432    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnngc\" (UniqueName: \"kubernetes.io/projected/2236c0f5-aab5-4979-8131-97477b58a186-kube-api-access-qnngc\") pod \"migrator-66df5d496b-sv7mk\" (UID: \"2236c0f5-aab5-4979-8131-97477b58a186\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"
Nov 01 08:32:07.053414 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.053369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"
Nov 01 08:32:07.100511 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.100486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:07.100641 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.100619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:07.100700 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:07.100648    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:07.100750 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:07.100699    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 08:32:07.100750 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:07.100723    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:08.100702734 +0000 UTC m=+40.368106242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:07.100750 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:07.100748    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:08.100732409 +0000 UTC m=+40.368135913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : configmap "kube-rbac-proxy" not found
Nov 01 08:32:07.459472 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.459446    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"]
Nov 01 08:32:07.531263 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.531225    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/1.log"
Nov 01 08:32:07.531684 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.531619    2579 scope.go:122] "RemoveContainer" containerID="92603f6a7e867a98d077b9a5c80c979a19f2d98a80e4b0aff735d8291e6bd7e0"
Nov 01 08:32:07.531868 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:07.531847    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jd869_openshift-console-operator(03c05080-9769-4bf4-ac9a-39b2c7301170)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" podUID="03c05080-9769-4bf4-ac9a-39b2c7301170"
Nov 01 08:32:07.533555 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.533527    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm" event={"ID":"8800f430-6279-4e64-aef8-7c75e914b72b","Type":"ContainerStarted","Data":"d05afd19946de6f0fb77245d99a52d4d036fe016b2ee00daa16f926e5b43815c"}
Nov 01 08:32:07.535705 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.535660    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mnsws" event={"ID":"44a34856-dd40-4408-9db2-67d075d9850a","Type":"ContainerStarted","Data":"24b72f54c33e53baa49ec45f52d11c6121993d115d91c1ce9aa3a5287b507003"}
Nov 01 08:32:07.536175 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.535957    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:07.537042 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.536983    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk" event={"ID":"2236c0f5-aab5-4979-8131-97477b58a186","Type":"ContainerStarted","Data":"a212704845bf9c462ac2a22a71abf7bed1d4b70f21b12779f6a389ad6c23062f"}
Nov 01 08:32:07.769329 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.769220    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm" podStartSLOduration=373.456384285 podStartE2EDuration="6m20.769204119s" totalImagesPullingTime="7.312819834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:07.768840663 +0000 UTC m=+40.036244176" watchObservedRunningTime="2026-11-01 08:32:07.769204119 +0000 UTC m=+40.036607633"
Nov 01 08:32:07.886044 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:07.885974    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mnsws" podStartSLOduration=34.758380036 podStartE2EDuration="39.885959849s" totalImagesPullingTime="5.127579813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:07.884686847 +0000 UTC m=+40.152090361" watchObservedRunningTime="2026-11-01 08:32:07.885959849 +0000 UTC m=+40.153363361"
Nov 01 08:32:08.111687 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:08.111647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:08.111860 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:08.111734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:08.111860 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:08.111856    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:08.112019 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:08.111908    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:10.111895133 +0000 UTC m=+42.379298623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:08.112282 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:08.112264    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 08:32:08.112337 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:08.112324    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:10.112306909 +0000 UTC m=+42.379710399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : configmap "kube-rbac-proxy" not found
Nov 01 08:32:09.022600 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:09.022552    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:09.023134 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:09.022622    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:09.023134 ip-10-0-143-244 kubenswrapper[2579]: I1101 08:32:09.023057    2579 scope.go:122] "RemoveContainer" containerID="92603f6a7e867a98d077b9a5c80c979a19f2d98a80e4b0aff735d8291e6bd7e0"
Nov 01 08:32:09.023324 ip-10-0-143-244 kubenswrapper[2579]: E1101 08:32:09.023293    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jd869_openshift-console-operator(03c05080-9769-4bf4-ac9a-39b2c7301170)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" podUID="03c05080-9769-4bf4-ac9a-39b2c7301170"
Nov 01 08:32:09.805760 ip-10-0-143-244 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 08:32:09.822445 ip-10-0-143-244 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 08:32:09.822619 ip-10-0-143-244 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 08:32:09.822667 ip-10-0-143-244 systemd[1]: kubelet.service: Consumed 3.141s CPU time.
Nov 01 08:32:09.836646 ip-10-0-143-244 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 08:32:09.950440 ip-10-0-143-244 kubenswrapper[4931]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:32:09.950440 ip-10-0-143-244 kubenswrapper[4931]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 08:32:09.950440 ip-10-0-143-244 kubenswrapper[4931]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:32:09.950440 ip-10-0-143-244 kubenswrapper[4931]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 08:32:09.952893 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952766    4931 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:09.952893 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952889    4931 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:09.952893 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952894    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:09.952893 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952898    4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952902    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952907    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952911    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952914    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952918    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952921    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952925    4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952928    4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952933    4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952945    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952949    4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952953    4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952957    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952961    4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952966    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952970    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952974    4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952977    4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952982    4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:09.953189 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952986    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952990    4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952994    4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.952998    4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953001    4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953006    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953010    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953014    4931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953039    4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953044    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953048    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953052    4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953060    4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953064    4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953069    4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953073    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953078    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953082    4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953086    4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:09.954061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953090    4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953094    4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953098    4931 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953102    4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953114    4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953121    4931 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953126    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953131    4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953135    4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953139    4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953144    4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953148    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953152    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953158    4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953162    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953167    4931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953171    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953175    4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953179    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953183    4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:09.954861 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953187    4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953191    4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953195    4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953199    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953203    4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953209    4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953214    4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953219    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953223    4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953227    4931 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953231    4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953236    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953240    4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953244    4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953248    4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953252    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953257    4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953262    4931 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953266    4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:09.955732 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953270    4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953274    4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953279    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953283    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953287    4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953291    4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953296    4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953300    4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953304    4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953311    4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953317    4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953322    4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953326    4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953331    4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953335    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953340    4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953344    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953348    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953356    4931 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:09.956630 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953360    4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953366    4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953372    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953377    4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953381    4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.953385    4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954661    4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954671    4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954676    4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954681    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954686    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954690    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954695    4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954700    4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954704    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954708    4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954713    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954717    4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954721    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:09.957517 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954725    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954729    4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954733    4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954738    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954741    4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954746    4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954750    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954755    4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954759    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954763    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954767    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954771    4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954776    4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954781    4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954785    4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954789    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954793    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954797    4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954802    4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954806    4931 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:09.958370 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954810    4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954814    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954818    4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954823    4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954827    4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954831    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954836    4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954840    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954844    4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954848    4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954852    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954856    4931 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954859    4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954863    4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954867    4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954870    4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954874    4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954878    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954884    4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:09.959232 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954888    4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954893    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954897    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954902    4931 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954906    4931 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954910    4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954914    4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954920    4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954924    4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954928    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954932    4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954936    4931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954940    4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954944    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954949    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954953    4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954959    4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954965    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954969    4931 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954974    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:09.959716 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954979    4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954983    4931 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954988    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954992    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.954996    4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955000    4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955004    4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955009    4931 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955013    4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955017    4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955041    4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955045    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955049    4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955053    4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955057    4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955062    4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955066    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955070    4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955075    4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955079    4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:09.960214 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955083    4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955089    4931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955093    4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955097    4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955101    4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955105    4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955109    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955113    4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955121    4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955128    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955133    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955138    4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955142    4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955154    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955956    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955964    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955969    4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955973    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955977    4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:09.960722 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955982    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955986    4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955990    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955995    4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.955999    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956003    4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956007    4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956012    4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956016    4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956040    4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956044    4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956048    4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956053    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956057    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956061    4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956065    4931 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956069    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956072    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956076    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956080    4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:09.961293 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956083    4931 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956087    4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956090    4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956094    4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956097    4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956101    4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956105    4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956109    4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956113    4931 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956116    4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956120    4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956123    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956127    4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956131    4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956135    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956139    4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956146    4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956150    4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956154    4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956159    4931 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:09.961763 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956163    4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956167    4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956171    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956175    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956179    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956183    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956188    4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956191    4931 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956196    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956201    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956205    4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956209    4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956214    4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956218    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956222    4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956226    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956230    4931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956234    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956238    4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956243    4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:09.962261 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956247    4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956251    4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956255    4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956259    4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956263    4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956267    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956271    4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956275    4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956282    4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956288    4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956292    4931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956296    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956300    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956304    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956309    4931 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956312    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956317    4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956321    4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956325    4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956329    4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:09.963172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956336    4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956342    4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956347    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956351    4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956358    4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956364    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956368    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956373    4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956378    4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956382    4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956386    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956390    4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956395    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956399    4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956404    4931 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956408    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956412    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956416    4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956421    4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:09.963798 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956426    4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.956433    4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956570    4931 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956585    4931 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956596    4931 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956603    4931 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956610    4931 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956614    4931 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956621    4931 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956627    4931 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956631    4931 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956636    4931 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956641    4931 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956646    4931 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956651    4931 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956655    4931 flags.go:64] FLAG: --cgroup-root=""
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956660    4931 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956665    4931 flags.go:64] FLAG: --client-ca-file=""
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956670    4931 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956674    4931 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956680    4931 flags.go:64] FLAG: --cluster-domain=""
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956685    4931 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956690    4931 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956695    4931 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 08:32:09.964301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956700    4931 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956706    4931 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956712    4931 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956717    4931 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956722    4931 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956727    4931 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956731    4931 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956739    4931 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956744    4931 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956750    4931 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956757    4931 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956762    4931 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956767    4931 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956771    4931 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956776    4931 flags.go:64] FLAG: --enable-server="true"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956781    4931 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956788    4931 flags.go:64] FLAG: --event-burst="100"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956793    4931 flags.go:64] FLAG: --event-qps="50"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956797    4931 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956802    4931 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956807    4931 flags.go:64] FLAG: --eviction-hard=""
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956813    4931 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956827    4931 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956832    4931 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956837    4931 flags.go:64] FLAG: --eviction-soft=""
Nov 01 08:32:09.964872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956842    4931 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956846    4931 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956851    4931 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956873    4931 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956877    4931 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956882    4931 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956886    4931 flags.go:64] FLAG: --feature-gates=""
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956893    4931 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956898    4931 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956903    4931 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956909    4931 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956914    4931 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956919    4931 flags.go:64] FLAG: --help="false"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956923    4931 flags.go:64] FLAG: --hostname-override="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956930    4931 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956935    4931 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956942    4931 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956947    4931 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956953    4931 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956958    4931 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956962    4931 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956967    4931 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956971    4931 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 08:32:09.965563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956976    4931 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956981    4931 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956986    4931 flags.go:64] FLAG: --kube-reserved=""
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956991    4931 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.956995    4931 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957000    4931 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957005    4931 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957009    4931 flags.go:64] FLAG: --lock-file=""
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957014    4931 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957018    4931 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957046    4931 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957055    4931 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957059    4931 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957064    4931 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957068    4931 flags.go:64] FLAG: --logging-format="text"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957073    4931 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957078    4931 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957082    4931 flags.go:64] FLAG: --manifest-url=""
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957087    4931 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957094    4931 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957099    4931 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957106    4931 flags.go:64] FLAG: --max-pods="110"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957111    4931 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957117    4931 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957122    4931 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 08:32:09.966153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957129    4931 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957134    4931 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957139    4931 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957144    4931 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957156    4931 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957161    4931 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957166    4931 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957170    4931 flags.go:64] FLAG: --pod-cidr=""
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957175    4931 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957179    4931 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957184    4931 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957189    4931 flags.go:64] FLAG: --port="10250"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957194    4931 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957199    4931 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f1abf34e2979c1a9"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957204    4931 flags.go:64] FLAG: --qos-reserved=""
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957209    4931 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957213    4931 flags.go:64] FLAG: --register-node="true"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957218    4931 flags.go:64] FLAG: --register-with-taints=""
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957224    4931 flags.go:64] FLAG: --registry-burst="10"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957229    4931 flags.go:64] FLAG: --registry-qps="5"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957234    4931 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957238    4931 flags.go:64] FLAG: --reserved-memory=""
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957245    4931 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957250    4931 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957254    4931 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957260    4931 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 08:32:09.966778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957264    4931 flags.go:64] FLAG: --runonce="false"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957269    4931 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957276    4931 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957281    4931 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957288    4931 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957293    4931 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957299    4931 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957304    4931 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957309    4931 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957313    4931 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957318    4931 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957323    4931 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957328    4931 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957333    4931 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957338    4931 flags.go:64] FLAG: --system-cgroups=""
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957342    4931 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957351    4931 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957356    4931 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957360    4931 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957367    4931 flags.go:64] FLAG: --tls-min-version=""
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957372    4931 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957376    4931 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957381    4931 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957385    4931 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957390    4931 flags.go:64] FLAG: --v="2"
Nov 01 08:32:09.967404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957403    4931 flags.go:64] FLAG: --version="false"
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957410    4931 flags.go:64] FLAG: --vmodule=""
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957416    4931 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.957422    4931 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957641    4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957648    4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957654    4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957660    4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957664    4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957669    4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957673    4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957681    4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957686    4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957692    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957696    4931 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957700    4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957705    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957709    4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957714    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957718    4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:09.967989 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957723    4931 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957727    4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957732    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957735    4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957740    4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957744    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957748    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957753    4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957757    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957761    4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957766    4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957770    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957774    4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957778    4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957782    4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957787    4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957791    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957795    4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957799    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:09.968583 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957803    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957808    4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957812    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957816    4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957823    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957829    4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957837    4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957842    4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957847    4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957851    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957855    4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957859    4931 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957864    4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957868    4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957872    4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957876    4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957880    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957885    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957889    4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:09.969175 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957896    4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957901    4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957908    4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957912    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957917    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957921    4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957926    4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957930    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957934    4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957938    4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957943    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957947    4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957951    4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957967    4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957971    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957975    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957979    4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957986    4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957991    4931 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.957999    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:09.969693 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958003    4931 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958008    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958012    4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958016    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958039    4931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958044    4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958048    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958052    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958057    4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958061    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958065    4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958069    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958073    4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958077    4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958081    4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958085    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958090    4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958094    4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958098    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:09.970264 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958102    4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958106    4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958110    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958114    4931 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958118    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958122    4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958126    4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958130    4931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958134    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958138    4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958145    4931 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958149    4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958155    4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.958201    4931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958257    4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958264    4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:09.970936 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958270    4931 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958274    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958278    4931 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958282    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958286    4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958290    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958295    4931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958299    4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958303    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958307    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958311    4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958315    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958319    4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958324    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958327    4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958332    4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958336    4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958340    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958344    4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:09.971418 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958349    4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958353    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958357    4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958361    4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958365    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958369    4931 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958374    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958380    4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958384    4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958388    4931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958394    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958399    4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958403    4931 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958407    4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958412    4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958416    4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958420    4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958425    4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958429    4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958433    4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:09.971901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958437    4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958441    4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958447    4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958451    4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958455    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958459    4931 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958463    4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958468    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958472    4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958476    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958480    4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958484    4931 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958488    4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958492    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958496    4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958500    4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958505    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958509    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958513    4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:09.972394 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958519    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958524    4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958528    4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958533    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958538    4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958542    4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958546    4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958551    4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958554    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958559    4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958563    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958567    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958571    4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958575    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958579    4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958583    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958590    4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958595    4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958600    4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:09.972901 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958606    4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958610    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958615    4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958619    4931 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958623    4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958627    4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958632    4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958636    4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958640    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958644    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958648    4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958655    4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958660    4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958667    4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958671    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958675    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958681    4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958686    4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958692    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:09.973388 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958696    4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958700    4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958704    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958707    4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958712    4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958716    4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958720    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.958723    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.963857    4931 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.963874    4931 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.963892    4931 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.963897    4931 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963961    4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963967    4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963970    4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963974    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963977    4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963979    4931 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963982    4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:09.973836 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963984    4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963987    4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963989    4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963991    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963994    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.963997    4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964000    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964003    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964006    4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964009    4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964011    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964014    4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964017    4931 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964019    4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964036    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964039    4931 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964042    4931 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964045    4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964048    4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:09.974328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964050    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964053    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964056    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964059    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964062    4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964065    4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964067    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964070    4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964072    4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964075    4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964077    4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964080    4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964082    4931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964085    4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964087    4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964090    4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964093    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964095    4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964098    4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964100    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:09.974771 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964104    4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964107    4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964109    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964112    4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964114    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964117    4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964119    4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964122    4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964124    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964127    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964129    4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964132    4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964134    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964136    4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964139    4931 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964141    4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964144    4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964146    4931 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964149    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964152    4931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:09.975251 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964154    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964157    4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964160    4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964162    4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964165    4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964167    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964170    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964172    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964175    4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964177    4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964179    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964182    4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964184    4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964188    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964191    4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964193    4931 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964196    4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964198    4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964201    4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964203    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:09.975707 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964206    4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964208    4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964210    4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964213    4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964217    4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964221    4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964224    4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964227    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964230    4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964232    4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964235    4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964237    4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964240    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964243    4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964247    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964250    4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964252    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964255    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964257    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:09.976195 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964259    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.964287    4931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964422    4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964427    4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964430    4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964432    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964435    4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964438    4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964441    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964443    4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964446    4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964448    4931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964451    4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964453    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964457    4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:09.976633 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964461    4931 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964464    4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964468    4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964471    4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964473    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964476    4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964478    4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964481    4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964483    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964486    4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964489    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964491    4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964494    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964496    4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964499    4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964501    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964504    4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964507    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964510    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:09.977009 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964512    4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964515    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964517    4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964519    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964522    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964524    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964527    4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964529    4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964532    4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964534    4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964537    4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964539    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964541    4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964544    4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964546    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964548    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964551    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964553    4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964555    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:09.977489 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964558    4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964560    4931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964563    4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964566    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964568    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964571    4931 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964573    4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964575    4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964578    4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964580    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964583    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964585    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964588    4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964590    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964593    4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964595    4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964597    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964600    4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964602    4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964605    4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:09.977998 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964607    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964610    4931 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964612    4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964615    4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964617    4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964619    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964622    4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964624    4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964626    4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964629    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964631    4931 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964634    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964636    4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964639    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964641    4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964644    4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964646    4931 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964648    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964651    4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964654    4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:09.978537 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964656    4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964658    4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964660    4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964663    4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964665    4931 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964667    4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964670    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964672    4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964675    4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964677    4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964679    4931 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964682    4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964684    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964687    4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964690    4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.964711    4931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 08:32:09.978997 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964742    4931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964746    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964749    4931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964751    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964754    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964757    4931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964760    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964762    4931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964765    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964767    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964770    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964772    4931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964775    4931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964777    4931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964780    4931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964782    4931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964785    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964787    4931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964790    4931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 08:32:09.979572 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964792    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964794    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964797    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964799    4931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964801    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964805    4931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964807    4931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964809    4931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964811    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964822    4931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964826    4931 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964829    4931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964831    4931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964834    4931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964836    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964839    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964841    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964844    4931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964846    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 08:32:09.980354 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964849    4931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964851    4931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964853    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964856    4931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964858    4931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964861    4931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964863    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964866    4931 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964868    4931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964871    4931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964873    4931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964875    4931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964878    4931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964900    4931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964903    4931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964907    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964910    4931 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964913    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964915    4931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964917    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 08:32:09.980971 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964920    4931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964923    4931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964925    4931 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964928    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964930    4931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964933    4931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964935    4931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964938    4931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964940    4931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964943    4931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964945    4931 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964947    4931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964955    4931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964957    4931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964960    4931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964963    4931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964965    4931 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964968    4931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964970    4931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964973    4931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 08:32:09.981487 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964975    4931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964980    4931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964982    4931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964985    4931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964988    4931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964990    4931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964993    4931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964996    4931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.964998    4931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965001    4931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965003    4931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965006    4931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965008    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965012    4931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965015    4931 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965019    4931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965043    4931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965047    4931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965050    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 08:32:09.982014 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965053    4931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965056    4931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965058    4931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965061    4931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965064    4931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965068    4931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965072    4931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965074    4931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:09.965077    4931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.965136    4931 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.966106    4931 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.966157    4931 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.966334    4931 server.go:1014] "Starting client certificate rotation"
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.966473    4931 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.966526    4931 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 08:26:28 +0000 UTC" deadline="2028-06-11 21:19:38.967583794 +0000 UTC"
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.966553    4931 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14124h47m29.001032492s"
Nov 01 08:32:09.982475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.966763    4931 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 08:32:09.982859 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.967843    4931 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 08:32:09.982859 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.969003    4931 log.go:25] "Validated CRI v1 runtime API"
Nov 01 08:32:09.982859 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.970665    4931 log.go:25] "Validated CRI v1 image API"
Nov 01 08:32:09.982859 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.971357    4931 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 08:32:09.982859 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.973204    4931 fs.go:135] Filesystem UUIDs: map[0f5fda05-0f32-4c8e-a4ba-e1a001e678c3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a9b872ad-a124-47a2-adba-9decb421333f:/dev/nvme0n1p3]
Nov 01 08:32:09.983011 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.973225    4931 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/097de667967bfd80c5a30fca7770d38bf52636bc413628f0db6c9fd1b1971dbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/097de667967bfd80c5a30fca7770d38bf52636bc413628f0db6c9fd1b1971dbd/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d999c2443cb79a08db1c59a7aacbef5988de6fa15087b5fde1bd683cea26b9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d999c2443cb79a08db1c59a7aacbef5988de6fa15087b5fde1bd683cea26b9e/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/128be4e192bcc15c75ab99af7509cf411896b091610e9e8cdbeb359d6bcc8db3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/128be4e192bcc15c75ab99af7509cf411896b091610e9e8cdbeb359d6bcc8db3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a64d543be1b8d2e4710491f63dd11bbaa22a4ab920dd1c3cd5001fa5a48c7cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a64d543be1b8d2e4710491f63dd11bbaa22a4ab920dd1c3cd5001fa5a48c7cb/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e2b5548b6a635aae3659ed3746e328f8dbd2b4f82ef17f43766d83c63a5ede4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e2b5548b6a635aae3659ed3746e328f8dbd2b4f82ef17f43766d83c63a5ede4/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3172d2f31dd0e27d233dc946cff3e5476f89e0650e43f02213088f214f22afe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3172d2f31dd0e27d233dc946cff3e5476f89e0650e43f02213088f214f22afe3/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/349ec0d7c233c56580a5860c15954b00d5f202a43428fd892362a68951f63791/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/349ec0d7c233c56580a5860c15954b00d5f202a43428fd892362a68951f63791/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/575e2349e4cf1c7c07bc74d785c8bade94d22027956713ebf950878a15305aa0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/575e2349e4cf1c7c07bc74d785c8bade94d22027956713ebf950878a15305aa0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b8f18bd537e454867d387c2eb85a01a2cd7ba5c5451ba7937d71dda0c597987/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b8f18bd537e454867d387c2eb85a01a2cd7ba5c5451ba7937d71dda0c597987/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d26fa0cc009b42bca2381546b1ad9583cc382aa66df9aaa1a530d4ef88e094d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d26fa0cc009b42bca2381546b1ad9583cc382aa66df9aaa1a530d4ef88e094d/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/763cdd6a30b86a98065bdd758b715a6495628f53b36ca23bacbc8f661684cbff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/763cdd6a30b86a98065bdd758b715a6495628f53b36ca23bacbc8f661684cbff/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d937ad8cda14b416ef7bdd404925f1b9e587015019b480934b0b9976104762a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d937ad8cda14b416ef7bdd404925f1b9e587015019b480934b0b9976104762a/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85a833ba2aa0af1bacf87e321ad076a890170df0ea9a2174f893a2c085d25654/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85a833ba2aa0af1bacf87e321ad076a890170df0ea9a2174f893a2c085d25654/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85dbc996b42e9b3598d3a35a65d153a8659205a89f560396fae4db22e066868e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85dbc996b42e9b3598d3a35a65d153a8659205a89f560396fae4db22e066868e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b821ad49a7a7feb12fa1a5446ad465996116eb48e8c1c6f6885d8ebf8a172bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b821ad49a7a7feb12fa1a5446ad465996116eb48e8c1c6f6885d8ebf8a172bd/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a212704845bf9c462ac2a22a71abf7bed1d4b70f21b12779f6a389ad6c23062f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a212704845bf9c462ac2a22a71abf7bed1d4b70f21b12779f6a389ad6c23062f/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5633325636bca4b69234a0fe51e659a0248a47f65b9df77c867137f94425011/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5633325636bca4b69234a0fe51e659a0248a47f65b9df77c867137f94425011/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8a9fac23e00f919297a254f0b8e41dbcca48a06164a7536e8eed8ac5d6396fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8a9fac23e00f919297a254f0b8e41dbcca48a06164a7536e8eed8ac5d6396fe/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba34e5192d359b06a15fce69f50ebebdc276cb8717a6d1bcb2779ef95cfa5ea0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba34e5192d359b06a15fce69f50ebebdc276cb8717a6d1bcb2779ef95cfa5ea0/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c061bb355e52e7c4853085bc2729e0732d97e901551b8b8281007f205f985446/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c061bb355e52e7c4853085bc2729e0732d97e901551b8b8281007f205f985446/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d35f7f8dab151f28f44aa9a14af073299847bda28c79fe56a0c340e594ed5373/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d35f7f8dab151f28f44aa9a14af073299847bda28c79fe56a0c340e594ed5373/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7c66d7176e870ead18d1eb0c2b2428066eefca505fb985b98e772c34faca2cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7c66d7176e870ead18d1eb0c2b2428066eefca505fb985b98e772c34faca2cf/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e005754778a8af730b692a521ffe6390c5d442ccb3f8b3fb4093c66edc2ad170/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e005754778a8af730b692a521ffe6390c5d442ccb3f8b3fb4093c66edc2ad170/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03c05080-9769-4bf4-ac9a-39b2c7301170/volumes/kubernetes.io~projected/kube-api-access-526c4:{mountpoint:/var/lib/kubelet/pods/03c05080-9769-4bf4-ac9a-39b2c7301170/volumes/kubernetes.io~projected/kube-api-access-526c4 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03c05080-9769-4bf4-ac9a-39b2c7301170/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/03c05080-9769-4bf4-ac9a-39b2c7301170/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08abdd05-c01c-4c69-ac14-330e24384e83/volumes/kubernetes.io~projected/kube-api-access-ndmds:{mountpoint:/var/lib/kubelet/pods/08abdd05-c01c-4c69-ac14-330e24384e83/volumes/kubernetes.io~projected/kube-api-access-ndmds major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb/volumes/kubernetes.io~projected/kube-api-access-xm7hr:{mountpoint:/var/lib/kubelet/pods/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb/volumes/kubernetes.io~projected/kube-api-access-xm7hr major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2236c0f5-aab5-4979-8131-97477b58a186/volumes/kubernetes.io~projected/kube-api-access-qnngc:{mountpoint:/var/lib/kubelet/pods/2236c0f5-aab5-4979-8131-97477b58a186/volumes/kubernetes.io~projected/kube-api-access-qnngc major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29a037fa-abd6-4bc2-9bc1-0d7da8429b09/volumes/kubernetes.io~projected/kube-api-access-522hk:{mountpoint:/var/lib/kubelet/pods/29a037fa-abd6-4bc2-9bc1-0d7da8429b09/volumes/kubernetes.io~projected/kube-api-access-522hk major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29a037fa-abd6-4bc2-9bc1-0d7da8429b09/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/29a037fa-abd6-4bc2-9bc1-0d7da8429b09/volumes/kubernetes.io~secret/cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e9c65da-47f6-494b-819e-34ee9896ff41/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2e9c65da-47f6-494b-819e-34ee9896ff41/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e/volumes/kubernetes.io~projected/kube-api-access-qf9jm:{mountpoint:/var/lib/kubelet/pods/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e/volumes/kubernetes.io~projected/kube-api-access-qf9jm major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~projected/kube-api-access-kgd5t:{mountpoint:/var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~projected/kube-api-access-kgd5t major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~secret/default-certificate major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~secret/stats-auth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44a34856-dd40-4408-9db2-67d075d9850a/volumes/kubernetes.io~projected/kube-api-access-q6qpf:{mountpoint:/var/lib/kubelet/pods/44a34856-dd40-4408-9db2-67d075d9850a/volumes/kubernetes.io~projected/kube-api-access-q6qpf major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46b41145-f5ea-4af3-9f72-b12fee0dd5e2/volumes/kubernetes.io~projected/kube-api-access-vgxtr:{mountpoint:/var/lib/kubelet/pods/46b41145-f5ea-4af3-9f72-b12fee0dd5e2/volumes/kubernetes.io~projected/kube-api-access-vgxtr major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f63fd1a-4af7-414d-aa39-4580e4e98d7f/volumes/kubernetes.io~projected/kube-api-access-hrt9t:{mountpoint:/var/lib/kubelet/pods/4f63fd1a-4af7-414d-aa39-4580e4e98d7f/volumes/kubernetes.io~projected/kube-api-access-hrt9t major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~projected/kube-api-access-fpw5p:{mountpoint:/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~projected/kube-api-access-fpw5p major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/608b447e-9dda-4503-bcef-26d6af784cfb/volumes/kubernetes.io~projected/kube-api-access-hmk98:{mountpoint:/var/lib/kubelet/pods/608b447e-9dda-4503-bcef-26d6af784cfb/volumes/kubernetes.io~projected/kube-api-access-hmk98 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67b4458b-d683-447a-a253-5b37d8531843/volumes/kubernetes.io~projected/kube-api-access-bt52q:{mountpoint:/var/lib/kubelet/pods/67b4458b-d683-447a-a253-5b37d8531843/volumes/kubernetes.io~projected/kube-api-access-bt52q major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~projected/kube-api-access-t5xkb:{mountpoint:/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~projected/kube-api-access-t5xkb major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f5bef45-538f-465e-85cf-84a97fc9ba78/volumes/kubernetes.io~projected/kube-api-access-ctpkb:{mountpoint:/var/lib/kubelet/pods/6f5bef45-538f-465e-85cf-84a97fc9ba78/volumes/kubernetes.io~projected/kube-api-access-ctpkb major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ea361a-8d47-4aee-8ed8-b742628152ba/volumes/kubernetes.io~projected/kube-api-access-wvttn:{mountpoint:/var/lib/kubelet/pods/81ea361a-8d47-4aee-8ed8-b742628152ba/volumes/kubernetes.io~projected/kube-api-access-wvttn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81ea361a-8d47-4aee-8ed8-b742628152ba/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/81ea361a-8d47-4aee-8ed8-b742628152ba/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8800f430-6279-4e64-aef8-7c75e914b72b/volumes/kubernetes.io~projected/kube-api-access-69gmq:{mountpoint:/var/lib/kubelet/pods/8800f430-6279-4e64-aef8-7c75e914b72b/volumes/kubernetes.io~projected/kube-api-access-69gmq major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89776b0e-b9bc-415b-8138-9ce3e9264a27/volumes/kubernetes.io~projected/kube-api-access-q5czf:{mountpoint:/var/lib/kubelet/pods/89776b0e-b9bc-415b-8138-9ce3e9264a27/volumes/kubernetes.io~projected/kube-api-access-q5czf major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900/volumes/kubernetes.io~projected/kube-api-access-66pcz:{mountpoint:/var/lib/kubelet/pods/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900/volumes/kubernetes.io~projected/kube-api-access-66pcz major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~projected/kube-api-access-bx899:{mountpoint:/var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~projected/kube-api-access-bx899 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae17519f-2301-43ce-ac7e-890042574c55/volumes/kubernetes.io~projected/kube-api-access-qssqc:{mountpoint:/var/lib/kubelet/pods/ae17519f-2301-43ce-ac7e-890042574c55/volumes/kubernetes.io~projected/kube-api-access-qssqc major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae17519f-2301-43ce-ac7e-890042574c55/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ae17519f-2301-43ce-ac7e-890042574c55/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afb159a0-e7ce-43e5-be5f-5260297660cb/volumes/kubernetes.io~projected/kube-api-access-2k5kb:{mountpoint:/var/lib/kubelet/pods/afb159a0-e7ce-43e5-be5f-5260297660cb/volumes/kubernetes.io~projected/kube-api-access-2k5kb major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd6ef98-6c6a-4687-bc85-4a540e06022a/volumes/kubernetes.io~projected/kube-api-access-2fkqz:{mountpoint:/var/lib/kubelet/pods/bcd6ef98-6c6a-4687-bc85-4a540e06022a/volumes/kubernetes.io~projected/kube-api-access-2fkqz major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f2d60e-838e-42df-8e42-58f036371709/volumes/kubernetes.io~projected/kube-api-access-wzwtt:{mountpoint:/var/lib/kubelet/pods/c6f2d60e-838e-42df-8e42-58f036371709/volumes/kubernetes.io~projected/kube-api-access-wzwtt major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de81ecfb-e731-40aa-9732-ca40772d9444/volumes/kubernetes.io~projected/kube-api-access-67hdn:{mountpoint:/var/lib/kubelet/pods/de81ecfb-e731-40aa-9732-ca40772d9444/volumes/kubernetes.io~projected/kube-api-access-67hdn major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e38ced3e-d98c-4df1-935f-a1cdc2abf313/volumes/kubernetes.io~projected/kube-api-access-89q2k:{mountpoint:/var/lib/kubelet/pods/e38ced3e-d98c-4df1-935f-a1cdc2abf313/volumes/kubernetes.io~projected/kube-api-access-89q2k major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5d1e97d-be9c-4f45-827c-19f789823a3b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e5d1e97d-be9c-4f45-827c-19f789823a3b/volumes/kubernetes.io~secret/agent-certs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volumes/kubernetes.io~projected/kube-api-access-s6997:{mountpoint:/var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volumes/kubernetes.io~projected/kube-api-access-s6997 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1f77890-a449-4958-9b83-ef93acbe2f58/volumes/kubernetes.io~projected/kube-api-access-7zfzs:{mountpoint:/var/lib/kubelet/pods/f1f77890-a449-4958-9b83-ef93acbe2f58/volumes/kubernetes.io~projected/kube-api-access-7zfzs major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9b2d6a0313cc49ec8321d4cc5c6d833a795235d3c6b1d99380f08a8e85392a95/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a29f62accb8fb6a0f99231a9d85afc17caa443dfc08e2610883cc516781f9b89/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c646954a22618cf0c3ab91d2953501d517dba20cc39ae0b76103a2640e756627/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/56e651817bda6bfc851bb197432182e5fb26a9cc15f5e681b16667f24b492c8f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/3a4776520d49bd0a6c43fcb28e22a7bdee765942cb8b0cdd4edf19aee0062faf/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c65f968e2a0ff0980b82c76dff7b93a46b894afe7e7818f4b9258f42b30567a8/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/76fbe929fc484648af150208b43e46e9bcf04c739a3c2cc5626e93b3fed6ed5a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1a8afa1f95af9950e0890e121e9aafb9e45822a9727d531572fe9ac479cfa604/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/3ff766e738ca265c7107fde5e54e00abbb6e7fba81a8683c0c956d2e3ffe059a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/94a670693d950056cdd669b4054c3499289e6e8eb8438b75866f7fa97ce5081a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/bef3be06ef19fc4e2c6abf9b4d8448170e032e6cd3725b14785376f3d3067249/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/8d34c4174c745ff0af38791821bb4847a8ad4365847bee575444506bcbd49799/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ade924d0c7741f0c362cee731e202dd9656d4a8856d9fe1aa9345073efd25d10/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2fb0a1cfdf1b2d18b226966c018dd13832672eca8e35bce3a4d25630c8cf360b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a7dbaf1f08d7f7ebf4dea5fed67e534b2fb5baea5430be5fd0280d1a08617fa4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1bbb1b87cde10d9fcc56da8f439410f3a88881d442e31a1c3afb9cd657e953da/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/656185e17cac0f4ca992aa38b48f02dbea524e9aa6a2c04f9e8082b4eb96509d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9b2880b38f4150da3ac524c54fca14283e4fce0b72eb8bdb1b8fb086fe54875b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b1aca2b97129791e94dc7b83512004c075d55ba4396ef75a889ea67d0bef7384/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0a5f6805ac85dbea8935184c22de5f079f360801612b6b0f0ddb6ddf7f4da8b3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b8561bea28489c56665fae549e533985670213a28dbc74e90b038df4ab136307/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/93728fed116daf2cd145bd2a420d254094881bad6b81058d0e37f41248940171/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/da33e1b6ac7eabaf39f853ee7cb52a091cfae7cf6fe28c3cbbe3a2ae00cf775a/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/682f155ea227de27837333b73720539b280ca28ae737fffeba84131d6d691c3b/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/8db938508a711c252af207da22b1a16a24ab423f9c8193178af8d6a929efe657/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9bb39857b7402c39e638151414245a23bbac432188b8419d41f4f81b362facdc/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/ac97e753b703924b8677e1a07918b5632fb9cac1186e65dd7475d5283534e552/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/36e81dbe28fe83cb06f8fe3a604186e46bbb2cb27516d9278ea294da93a08395/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2e7827d10943621a4df027e64717d556aafaa3b132ced99786b21b267f6bd088/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/7c310c0e64009803a4f720ddaac214b492292c3b8fbbe88db9dea95e445bd129/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/8caf9bb3e16ea29a0e8eae3bb83b3e5e61c163829063b56e459216f08a8a5267/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/d95ecbaca11c679b86d765fbe024c85af0de9bd4c41d7bd0e870c3f4b86867d6/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/5344a1eb259753c1bfbeec28ba3333ee9f30ca39e9c1404ead3fc32edb8a9e4c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/75b6fe6384b85e8c16023c2b3e320091b9152522650560e4cda9242627d2d6b1/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/34469f8e46791a0880b1e7aa647e1a17e893643fde84d43564b8e513ac2688b4/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/0e04f2ab7d0680c88932eb116a4591deaf71b798e59191a1ee2ac134aa147761/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/109530f1621cca4b9358f60cdbcdeb909036dc20f973aadf3837aa7d251577d6/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/fd0c43b86c367ca24f9b393a0dbed5a141573805c6d9515db0884e4f8149019d/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/befe05483bd831a233f094a05ff6e0f07cbaf5702ba613ed33db40a5d5f3eced/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/c475cacbd601cb9f45bdb81046bfe5e4b07e6bc254bf65c9e11820a8eba8f287/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/5d39560fb328001a1e84758f70770cb945d003de122b8626093966f92f90cabb/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/7d19983a212aa9c860be284bdc493654b386e0c199cc3f0ee95bbb74daafb661/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/a1a212b226eedde2f56126241de9001d316228d2c974077e49a6a39418cfe376/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/f5b03e3c1620e117c1c9441c53bccdcac3518cee97a14ea62f86457802410d41/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/288bdf1fabae0bcc161c98fbbb53951e3593111b3de6de1bfda60d361f1008bd/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/af6d7aa205109356cd7d609a40989fb161d75606cb361f72bf8b8cca8987f61b/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b48289ae4f73eab2508976d7940268e904b8e18d0cdc56ae5d412c9dde2c0eee/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f8cd0d571d4846559b1afca093278355de09b2592f19bb30f234b3795d625278/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/88310b5427aa4ab9baf4f395ee988fc2e421d59ae26e04cd6f100a495af91caa/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/67151b8c4a61bd8c48e7ceb15173aa5f5e5522bd410939daa056136b28a1ccc4/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a13947539a365b3257db7c4a210f3b12ac93701394befb095eab994aa4db6e79/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/f38e0a92de87da6a0f85e210d2b1f91ba67fe8362d055bccedf61ca2ad9b81ef/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e179e8c6b1b1f5f72e570450410fef84e1351368e52e836d7d72b312c53aaece/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/cebf118f624cdb475257294c4c62cc4f5e95d2943f860a6a8d7b897361c5f9ed/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b9ef9b2db4461030f5d8d307505e02bbb6c7498feadbf6e0e6b7406f08619606/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 08:32:09.983273 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.982525    4931 manager.go:217] Machine: {Timestamp:2026-11-01 08:32:09.982082433 +0000 UTC m=+0.096955391 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3169165 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f054e35028055c95c233acd14d966 SystemUUID:ec2f054e-3502-8055-c95c-233acd14d966 BootID:54992d16-ec53-4852-861d-2d47bab89ef7 Filesystems:[{Device:/var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~projected/kube-api-access-kgd5t DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b8f18bd537e454867d387c2eb85a01a2cd7ba5c5451ba7937d71dda0c597987/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/097de667967bfd80c5a30fca7770d38bf52636bc413628f0db6c9fd1b1971dbd/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a212704845bf9c462ac2a22a71abf7bed1d4b70f21b12779f6a389ad6c23062f/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e2b5548b6a635aae3659ed3746e328f8dbd2b4f82ef17f43766d83c63a5ede4/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/763cdd6a30b86a98065bdd758b715a6495628f53b36ca23bacbc8f661684cbff/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e/volumes/kubernetes.io~projected/kube-api-access-qf9jm DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/128be4e192bcc15c75ab99af7509cf411896b091610e9e8cdbeb359d6bcc8db3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ca1667a-32ed-4777-8755-ec7e72add722/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e9c65da-47f6-494b-819e-34ee9896ff41/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2236c0f5-aab5-4979-8131-97477b58a186/volumes/kubernetes.io~projected/kube-api-access-qnngc DeviceMajor:0 DeviceMinor:324 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8a9fac23e00f919297a254f0b8e41dbcca48a06164a7536e8eed8ac5d6396fe/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd6ef98-6c6a-4687-bc85-4a540e06022a/volumes/kubernetes.io~projected/kube-api-access-2fkqz DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f2d60e-838e-42df-8e42-58f036371709/volumes/kubernetes.io~projected/kube-api-access-wzwtt DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85a833ba2aa0af1bacf87e321ad076a890170df0ea9a2174f893a2c085d25654/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c061bb355e52e7c4853085bc2729e0732d97e901551b8b8281007f205f985446/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29a037fa-abd6-4bc2-9bc1-0d7da8429b09/volumes/kubernetes.io~projected/kube-api-access-522hk DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8800f430-6279-4e64-aef8-7c75e914b72b/volumes/kubernetes.io~projected/kube-api-access-69gmq DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/349ec0d7c233c56580a5860c15954b00d5f202a43428fd892362a68951f63791/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d937ad8cda14b416ef7bdd404925f1b9e587015019b480934b0b9976104762a/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81ea361a-8d47-4aee-8ed8-b742628152ba/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44a34856-dd40-4408-9db2-67d075d9850a/volumes/kubernetes.io~projected/kube-api-access-q6qpf DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f63fd1a-4af7-414d-aa39-4580e4e98d7f/volumes/kubernetes.io~projected/kube-api-access-hrt9t DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ae17519f-2301-43ce-ac7e-890042574c55/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5d1e97d-be9c-4f45-827c-19f789823a3b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85dbc996b42e9b3598d3a35a65d153a8659205a89f560396fae4db22e066868e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~projected/kube-api-access-fpw5p DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03c05080-9769-4bf4-ac9a-39b2c7301170/volumes/kubernetes.io~projected/kube-api-access-526c4 DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d26fa0cc009b42bca2381546b1ad9583cc382aa66df9aaa1a530d4ef88e094d/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5633325636bca4b69234a0fe51e659a0248a47f65b9df77c867137f94425011/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67b4458b-d683-447a-a253-5b37d8531843/volumes/kubernetes.io~projected/kube-api-access-bt52q DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08abdd05-c01c-4c69-ac14-330e24384e83/volumes/kubernetes.io~projected/kube-api-access-ndmds DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46b41145-f5ea-4af3-9f72-b12fee0dd5e2/volumes/kubernetes.io~projected/kube-api-access-vgxtr DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/608b447e-9dda-4503-bcef-26d6af784cfb/volumes/kubernetes.io~projected/kube-api-access-hmk98 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:237 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e005754778a8af730b692a521ffe6390c5d442ccb3f8b3fb4093c66edc2ad170/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e38ced3e-d98c-4df1-935f-a1cdc2abf313/volumes/kubernetes.io~projected/kube-api-access-89q2k DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de81ecfb-e731-40aa-9732-ca40772d9444/volumes/kubernetes.io~projected/kube-api-access-67hdn DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a108cdc6-7b79-4e7a-8b7d-7824406258e3/volumes/kubernetes.io~projected/kube-api-access-bx899 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7c66d7176e870ead18d1eb0c2b2428066eefca505fb985b98e772c34faca2cf/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/575e2349e4cf1c7c07bc74d785c8bade94d22027956713ebf950878a15305aa0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b821ad49a7a7feb12fa1a5446ad465996116eb48e8c1c6f6885d8ebf8a172bd/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81ea361a-8d47-4aee-8ed8-b742628152ba/volumes/kubernetes.io~projected/kube-api-access-wvttn DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d35f7f8dab151f28f44aa9a14af073299847bda28c79fe56a0c340e594ed5373/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ae17519f-2301-43ce-ac7e-890042574c55/volumes/kubernetes.io~projected/kube-api-access-qssqc DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/afb159a0-e7ce-43e5-be5f-5260297660cb/volumes/kubernetes.io~projected/kube-api-access-2k5kb DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3172d2f31dd0e27d233dc946cff3e5476f89e0650e43f02213088f214f22afe3/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d999c2443cb79a08db1c59a7aacbef5988de6fa15087b5fde1bd683cea26b9e/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1f77890-a449-4958-9b83-ef93acbe2f58/volumes/kubernetes.io~projected/kube-api-access-7zfzs DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f5bef45-538f-465e-85cf-84a97fc9ba78/volumes/kubernetes.io~projected/kube-api-access-ctpkb DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900/volumes/kubernetes.io~projected/kube-api-access-66pcz DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb/volumes/kubernetes.io~projected/kube-api-access-xm7hr DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03c05080-9769-4bf4-ac9a-39b2c7301170/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba34e5192d359b06a15fce69f50ebebdc276cb8717a6d1bcb2779ef95cfa5ea0/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29a037fa-abd6-4bc2-9bc1-0d7da8429b09/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes/kubernetes.io~projected/kube-api-access-t5xkb DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89776b0e-b9bc-415b-8138-9ce3e9264a27/volumes/kubernetes.io~projected/kube-api-access-q5czf DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f13e4499-bb2f-4e37-bd37-52f4bdad02dd/volumes/kubernetes.io~projected/kube-api-access-s6997 DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a64d543be1b8d2e4710491f63dd11bbaa22a4ab920dd1c3cd5001fa5a48c7cb/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:097de667967bfd8 MacAddress:62:46:65:7f:64:c9 Speed:10000 Mtu:8901} {Name:0d999c2443cb79a MacAddress:06:94:3d:7d:18:f6 Speed:10000 Mtu:8901} {Name:1e2b5548b6a635a MacAddress:46:30:6e:e5:67:f4 Speed:10000 Mtu:8901} {Name:349ec0d7c233c56 MacAddress:36:7a:6a:a5:44:bc Speed:10000 Mtu:8901} {Name:5b8f18bd537e454 MacAddress:22:24:e9:7c:9c:b0 Speed:10000 Mtu:8901} {Name:5d26fa0cc009b42 MacAddress:86:eb:b7:0b:64:8c Speed:10000 Mtu:8901} {Name:763cdd6a30b86a9 MacAddress:4e:c9:de:77:88:bd Speed:10000 Mtu:8901} {Name:a212704845bf9c4 MacAddress:92:0c:63:c5:ad:77 Speed:10000 Mtu:8901} {Name:b5633325636bca4 MacAddress:ba:83:ea:1e:c8:eb Speed:10000 Mtu:8901} {Name:ba34e5192d359b0 MacAddress:5a:84:f0:1a:12:f1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c6:16:03:2d Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:6d:44:df:cc:6d Speed:0 Mtu:8901} {Name:c061bb355e52e7c MacAddress:0a:8c:d7:b1:03:98 Speed:10000 Mtu:8901} {Name:d35f7f8dab151f2 MacAddress:3e:84:6a:d6:24:67 Speed:10000 Mtu:8901} {Name:e005754778a8af7 MacAddress:e2:a0:5c:b6:e5:fd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c6:16:03:2d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2a:67:28:3c:00:34 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:25:ce:f2:3f:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 08:32:09.983273 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.982859    4931 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 08:32:09.983273 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.982901    4931 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 08:32:09.983273 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983191    4931 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 08:32:09.983526 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983210    4931 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-244.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 08:32:09.983526 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983332    4931 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 08:32:09.983526 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983340    4931 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 08:32:09.983526 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983350    4931 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 08:32:09.983526 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983368    4931 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 08:32:09.983526 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983386    4931 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 08:32:09.983708 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983530    4931 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 08:32:09.983708 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983647    4931 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 08:32:09.983708 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983679    4931 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 08:32:09.983708 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983692    4931 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 08:32:09.983708 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983701    4931 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 08:32:09.983708 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983709    4931 kubelet.go:405] "Adding apiserver pod source"
Nov 01 08:32:09.983953 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.983717    4931 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 08:32:09.984546 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.984528    4931 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 08:32:09.984546 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.984545    4931 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 08:32:09.985048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.985016    4931 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 08:32:09.997269 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997253    4931 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 08:32:09.997269 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997267    4931 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 08:32:09.997381 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997369    4931 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 08:32:09.997567 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997556    4931 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 08:32:09.997607 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997586    4931 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 08:32:09.997708 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997696    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 08:32:09.997741 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997718    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 08:32:09.997741 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997734    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 08:32:09.997813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997748    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 08:32:09.997813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997761    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 08:32:09.997813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997774    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 08:32:09.997813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997788    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 08:32:09.997813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997801    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 08:32:09.997944 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997818    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 08:32:09.997944 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997829    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 08:32:09.997944 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997838    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 08:32:09.997944 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997847    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 08:32:09.997944 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997873    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 08:32:09.997944 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:09.997883    4931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 08:32:10.001524 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.001503    4931 server.go:1263] "Started kubelet"
Nov 01 08:32:10.002111 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.001594    4931 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 08:32:10.002111 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.001691    4931 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 08:32:10.002111 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.001605    4931 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 08:32:10.002271 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.002243    4931 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 08:32:10.002148 ip-10-0-143-244 systemd[1]: Started Kubernetes Kubelet.
Nov 01 08:32:10.007516 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.004810    4931 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 08:32:10.008265 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.008245    4931 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 08:32:10.009263 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.009241    4931 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 08:32:10.009348 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.009297    4931 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 08:32:10.009348 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.009306    4931 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 08:32:10.009441 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.009388    4931 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:29 +0000 UTC" deadline="2028-03-27 05:44:35.998850111 +0000 UTC"
Nov 01 08:32:10.009441 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.009405    4931 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12285h12m25.9894481s"
Nov 01 08:32:10.009441 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.009432    4931 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.us-west-2.compute.internal\" not found"
Nov 01 08:32:10.009583 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.009525    4931 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 08:32:10.011697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.010940    4931 factory.go:153] Registering CRI-O factory
Nov 01 08:32:10.011697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.010961    4931 factory.go:223] Registration of the crio container factory successfully
Nov 01 08:32:10.011697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.011042    4931 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 08:32:10.011697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.011054    4931 factory.go:55] Registering systemd factory
Nov 01 08:32:10.011697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.011061    4931 factory.go:223] Registration of the systemd container factory successfully
Nov 01 08:32:10.011697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.011078    4931 factory.go:103] Registering Raw factory
Nov 01 08:32:10.011697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.011088    4931 manager.go:1199] Started watching for new ooms in manager
Nov 01 08:32:10.011697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.011472    4931 manager.go:319] Starting recovery of all containers
Nov 01 08:32:10.012233 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012206    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" volumeName="kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-binary-copy" seLinuxMountContext=""
Nov 01 08:32:10.012289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012233    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13e4499-bb2f-4e37-bd37-52f4bdad02dd" volumeName="kubernetes.io/secret/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 08:32:10.012289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012243    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08abdd05-c01c-4c69-ac14-330e24384e83" volumeName="kubernetes.io/empty-dir/08abdd05-c01c-4c69-ac14-330e24384e83-data-volume" seLinuxMountContext=""
Nov 01 08:32:10.012289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012251    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" volumeName="kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 08:32:10.012289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012259    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ca1667a-32ed-4777-8755-ec7e72add722" volumeName="kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-default-certificate" seLinuxMountContext=""
Nov 01 08:32:10.012289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012267    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ea361a-8d47-4aee-8ed8-b742628152ba" volumeName="kubernetes.io/projected/81ea361a-8d47-4aee-8ed8-b742628152ba-kube-api-access-wvttn" seLinuxMountContext=""
Nov 01 08:32:10.012289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012274    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" volumeName="kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca" seLinuxMountContext=""
Nov 01 08:32:10.012289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012283    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ca1667a-32ed-4777-8755-ec7e72add722" volumeName="kubernetes.io/projected/3ca1667a-32ed-4777-8755-ec7e72add722-kube-api-access-kgd5t" seLinuxMountContext=""
Nov 01 08:32:10.012289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012290    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89776b0e-b9bc-415b-8138-9ce3e9264a27" volumeName="kubernetes.io/configmap/89776b0e-b9bc-415b-8138-9ce3e9264a27-iptables-alerter-script" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012300    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89776b0e-b9bc-415b-8138-9ce3e9264a27" volumeName="kubernetes.io/projected/89776b0e-b9bc-415b-8138-9ce3e9264a27-kube-api-access-q5czf" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012307    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae17519f-2301-43ce-ac7e-890042574c55" volumeName="kubernetes.io/configmap/ae17519f-2301-43ce-ac7e-890042574c55-config" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012313    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f2d60e-838e-42df-8e42-58f036371709" volumeName="kubernetes.io/projected/c6f2d60e-838e-42df-8e42-58f036371709-kube-api-access-wzwtt" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012320    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5d1e97d-be9c-4f45-827c-19f789823a3b" volumeName="kubernetes.io/configmap/e5d1e97d-be9c-4f45-827c-19f789823a3b-konnectivity-ca" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012331    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13e4499-bb2f-4e37-bd37-52f4bdad02dd" volumeName="kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-env-overrides" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012340    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03c05080-9769-4bf4-ac9a-39b2c7301170" volumeName="kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-config" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012359    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608b447e-9dda-4503-bcef-26d6af784cfb" volumeName="kubernetes.io/empty-dir/608b447e-9dda-4503-bcef-26d6af784cfb-tmp-dir" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012366    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ddfd7c8-e096-40dc-8e3d-a359cb566176" volumeName="kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012373    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ddfd7c8-e096-40dc-8e3d-a359cb566176" volumeName="kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012381    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ea361a-8d47-4aee-8ed8-b742628152ba" volumeName="kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-tmp" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012388    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1f77890-a449-4958-9b83-ef93acbe2f58" volumeName="kubernetes.io/projected/f1f77890-a449-4958-9b83-ef93acbe2f58-kube-api-access-7zfzs" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012396    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2023ca6f-9936-4e4c-80a1-e33b47fa7dcb" volumeName="kubernetes.io/configmap/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-config" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012403    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29a037fa-abd6-4bc2-9bc1-0d7da8429b09" volumeName="kubernetes.io/projected/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-kube-api-access-522hk" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012409    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" volumeName="kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012415    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a108cdc6-7b79-4e7a-8b7d-7824406258e3" volumeName="kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-tmp" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012422    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae17519f-2301-43ce-ac7e-890042574c55" volumeName="kubernetes.io/projected/ae17519f-2301-43ce-ac7e-890042574c55-kube-api-access-qssqc" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012428    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13e4499-bb2f-4e37-bd37-52f4bdad02dd" volumeName="kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-config" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012446    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03c05080-9769-4bf4-ac9a-39b2c7301170" volumeName="kubernetes.io/secret/03c05080-9769-4bf4-ac9a-39b2c7301170-serving-cert" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012454    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f63fd1a-4af7-414d-aa39-4580e4e98d7f" volumeName="kubernetes.io/projected/4f63fd1a-4af7-414d-aa39-4580e4e98d7f-kube-api-access-hrt9t" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012473    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ea361a-8d47-4aee-8ed8-b742628152ba" volumeName="kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-service-ca-bundle" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012480    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ea361a-8d47-4aee-8ed8-b742628152ba" volumeName="kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-snapshots" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012487    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd6ef98-6c6a-4687-bc85-4a540e06022a" volumeName="kubernetes.io/projected/bcd6ef98-6c6a-4687-bc85-4a540e06022a-kube-api-access-2fkqz" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012497    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46b41145-f5ea-4af3-9f72-b12fee0dd5e2" volumeName="kubernetes.io/projected/46b41145-f5ea-4af3-9f72-b12fee0dd5e2-kube-api-access-vgxtr" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012519    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2023ca6f-9936-4e4c-80a1-e33b47fa7dcb" volumeName="kubernetes.io/projected/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-kube-api-access-xm7hr" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012530    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2236c0f5-aab5-4979-8131-97477b58a186" volumeName="kubernetes.io/projected/2236c0f5-aab5-4979-8131-97477b58a186-kube-api-access-qnngc" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012537    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ca1667a-32ed-4777-8755-ec7e72add722" volumeName="kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-stats-auth" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012543    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ddfd7c8-e096-40dc-8e3d-a359cb566176" volumeName="kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012551    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de81ecfb-e731-40aa-9732-ca40772d9444" volumeName="kubernetes.io/projected/de81ecfb-e731-40aa-9732-ca40772d9444-kube-api-access-67hdn" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012558    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03c05080-9769-4bf4-ac9a-39b2c7301170" volumeName="kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-trusted-ca" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012569    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05aad278-6386-4a54-89c9-c143151e86cf" volumeName="kubernetes.io/configmap/05aad278-6386-4a54-89c9-c143151e86cf-nginx-conf" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012579    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2023ca6f-9936-4e4c-80a1-e33b47fa7dcb" volumeName="kubernetes.io/secret/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-serving-cert" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012587    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e9c65da-47f6-494b-819e-34ee9896ff41" volumeName="kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012593    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" volumeName="kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012600    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900" volumeName="kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-daemon-config" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012606    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ddfd7c8-e096-40dc-8e3d-a359cb566176" volumeName="kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012615    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f2d60e-838e-42df-8e42-58f036371709" volumeName="kubernetes.io/configmap/c6f2d60e-838e-42df-8e42-58f036371709-config-volume" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012623    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a108cdc6-7b79-4e7a-8b7d-7824406258e3" volumeName="kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-tuned" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012629    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ea361a-8d47-4aee-8ed8-b742628152ba" volumeName="kubernetes.io/secret/81ea361a-8d47-4aee-8ed8-b742628152ba-serving-cert" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012636    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8800f430-6279-4e64-aef8-7c75e914b72b" volumeName="kubernetes.io/projected/8800f430-6279-4e64-aef8-7c75e914b72b-kube-api-access-69gmq" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012644    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900" volumeName="kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cni-binary-copy" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012650    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" volumeName="kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012657    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" volumeName="kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012665    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29a037fa-abd6-4bc2-9bc1-0d7da8429b09" volumeName="kubernetes.io/secret/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-cert" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012672    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" volumeName="kubernetes.io/projected/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-kube-api-access-qf9jm" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012679    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de81ecfb-e731-40aa-9732-ca40772d9444" volumeName="kubernetes.io/configmap/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-config" seLinuxMountContext=""
Nov 01 08:32:10.012667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012696    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13e4499-bb2f-4e37-bd37-52f4bdad02dd" volumeName="kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012704    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44a34856-dd40-4408-9db2-67d075d9850a" volumeName="kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012727    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" volumeName="kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012735    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ddfd7c8-e096-40dc-8e3d-a359cb566176" volumeName="kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012747    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f2d60e-838e-42df-8e42-58f036371709" volumeName="kubernetes.io/empty-dir/c6f2d60e-838e-42df-8e42-58f036371709-tmp-dir" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012754    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae17519f-2301-43ce-ac7e-890042574c55" volumeName="kubernetes.io/secret/ae17519f-2301-43ce-ac7e-890042574c55-serving-cert" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012760    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13e4499-bb2f-4e37-bd37-52f4bdad02dd" volumeName="kubernetes.io/projected/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-kube-api-access-s6997" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012767    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1f77890-a449-4958-9b83-ef93acbe2f58" volumeName="kubernetes.io/configmap/f1f77890-a449-4958-9b83-ef93acbe2f58-telemetry-config" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012773    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03c05080-9769-4bf4-ac9a-39b2c7301170" volumeName="kubernetes.io/projected/03c05080-9769-4bf4-ac9a-39b2c7301170-kube-api-access-526c4" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012782    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900" volumeName="kubernetes.io/projected/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-kube-api-access-66pcz" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012792    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a108cdc6-7b79-4e7a-8b7d-7824406258e3" volumeName="kubernetes.io/projected/a108cdc6-7b79-4e7a-8b7d-7824406258e3-kube-api-access-bx899" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012801    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e38ced3e-d98c-4df1-935f-a1cdc2abf313" volumeName="kubernetes.io/configmap/e38ced3e-d98c-4df1-935f-a1cdc2abf313-serviceca" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012808    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08abdd05-c01c-4c69-ac14-330e24384e83" volumeName="kubernetes.io/projected/08abdd05-c01c-4c69-ac14-330e24384e83-kube-api-access-ndmds" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012822    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" volumeName="kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012833    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608b447e-9dda-4503-bcef-26d6af784cfb" volumeName="kubernetes.io/projected/608b447e-9dda-4503-bcef-26d6af784cfb-kube-api-access-hmk98" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012840    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ddfd7c8-e096-40dc-8e3d-a359cb566176" volumeName="kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012847    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ddfd7c8-e096-40dc-8e3d-a359cb566176" volumeName="kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012874    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" volumeName="kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012886    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b4458b-d683-447a-a253-5b37d8531843" volumeName="kubernetes.io/projected/67b4458b-d683-447a-a253-5b37d8531843-kube-api-access-bt52q" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012894    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f5bef45-538f-465e-85cf-84a97fc9ba78" volumeName="kubernetes.io/projected/6f5bef45-538f-465e-85cf-84a97fc9ba78-kube-api-access-ctpkb" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012900    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81ea361a-8d47-4aee-8ed8-b742628152ba" volumeName="kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012908    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afb159a0-e7ce-43e5-be5f-5260297660cb" volumeName="kubernetes.io/projected/afb159a0-e7ce-43e5-be5f-5260297660cb-kube-api-access-2k5kb" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012915    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e38ced3e-d98c-4df1-935f-a1cdc2abf313" volumeName="kubernetes.io/projected/e38ced3e-d98c-4df1-935f-a1cdc2abf313-kube-api-access-89q2k" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012921    4931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5d1e97d-be9c-4f45-827c-19f789823a3b" volumeName="kubernetes.io/secret/e5d1e97d-be9c-4f45-827c-19f789823a3b-agent-certs" seLinuxMountContext=""
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012929    4931 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 08:32:10.014761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.012934    4931 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 08:32:10.015876 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.015238    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/1.log"
Nov 01 08:32:10.018804 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.018655    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/1.log"
Nov 01 08:32:10.021565 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.021546    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:32:10.028952 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.028932    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:32:10.038360 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.038332    4931 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 08:32:10.039867 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.039848    4931 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 08:32:10.039867 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.039866    4931 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 08:32:10.040008 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.039884    4931 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 08:32:10.040357 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.040298    4931 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 08:32:10.055382 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.055330    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/1.log"
Nov 01 08:32:10.055382 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.055364    4931 generic.go:356] "Generic (PLEG): container finished" podID="03c05080-9769-4bf4-ac9a-39b2c7301170" containerID="92603f6a7e867a98d077b9a5c80c979a19f2d98a80e4b0aff735d8291e6bd7e0" exitCode=255
Nov 01 08:32:10.059066 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.059043    4931 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="818f81de48788899a0f031371bc3222d185469b68f1c169d83b7bea1271c9779" exitCode=0
Nov 01 08:32:10.059137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.059069    4931 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="d7f66ef7609d20cd261c7568e05fa2bdd3a904d1cd15891b14cd03866d29c4d9" exitCode=0
Nov 01 08:32:10.059137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.059079    4931 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="2d05e1b1fb8d7e23b7d72cb03c7aeacb0257df61f11806680512ae8e0a0b93f1" exitCode=0
Nov 01 08:32:10.059137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.059090    4931 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="86aba9dbf1dea8b53a55f25654b12d9ae0b832b31ecdff497154df7dbfeac3c9" exitCode=0
Nov 01 08:32:10.059137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.059101    4931 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="e62fce31f069de95945e62886490d4b998204428007322b02685b86241415e55" exitCode=0
Nov 01 08:32:10.059137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.059111    4931 generic.go:356] "Generic (PLEG): container finished" podID="3a214d76-85af-4fd5-9ccd-6cdce46fdd6e" containerID="fa3a14ea5192eb910d78da7a942eabe95c683f499048911036385ca5a19651ba" exitCode=0
Nov 01 08:32:10.064672 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.064646    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:32:10.064981 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.064959    4931 generic.go:356] "Generic (PLEG): container finished" podID="f13e4499-bb2f-4e37-bd37-52f4bdad02dd" containerID="82439b0904e4b96abfab98c648dc1bdf2c255be7a6e484c63c605c5ae43cad83" exitCode=1
Nov 01 08:32:10.069888 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.069866    4931 generic.go:356] "Generic (PLEG): container finished" podID="ca0fe841d453cc229ff1f7a2450a3ba4" containerID="22ff066c1e5e1aa34801ff452c3132f8487894816f1ecf19854f9943f6686135" exitCode=0
Nov 01 08:32:10.100125 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.100102    4931 manager.go:324] Recovery completed
Nov 01 08:32:10.110491 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.110466    4931 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.us-west-2.compute.internal\" not found"
Nov 01 08:32:10.116662 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.116646    4931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:10.118641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.118622    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:10.118709 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.118661    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:10.118709 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.118673    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:10.121349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121329    4931 cpu_manager.go:226] "Starting" policy="none"
Nov 01 08:32:10.121349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121344    4931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:10.121485 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121345    4931 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 08:32:10.121485 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121440    4931 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 08:32:10.121573 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121562    4931 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 08:32:10.121636 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121574    4931 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 08:32:10.121636 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121591    4931 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 08:32:10.121636 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121597    4931 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 08:32:10.121636 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121603    4931 policy_none.go:50] "Start"
Nov 01 08:32:10.121636 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121612    4931 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 08:32:10.121636 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121621    4931 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 08:32:10.121820 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121733    4931 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 08:32:10.121820 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121740    4931 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 08:32:10.121820 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.121746    4931 policy_none.go:44] "Start"
Nov 01 08:32:10.122344 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.122329    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:10.122401 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.122357    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:10.122401 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.122367    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:10.125856 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.125832    4931 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 08:32:10.128575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.128562    4931 manager.go:341] "Starting Device Plugin manager"
Nov 01 08:32:10.128669 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.128585    4931 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 08:32:10.128669 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.128594    4931 server.go:87] "Starting device plugin registration server"
Nov 01 08:32:10.128878 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.128847    4931 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 08:32:10.128934 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.128878    4931 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 08:32:10.128982 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.128933    4931 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 08:32:10.129473 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.129421    4931 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 08:32:10.129473 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.129471    4931 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 08:32:10.129614 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.129479    4931 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 08:32:10.129665 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.129548    4931 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T08:32:10.129459439Z","UUID":"2db990e3-234a-47e0-ad12-3ad7288c9e55","Handler":null,"Name":"","Endpoint":""}
Nov 01 08:32:10.130968 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.130944    4931 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 08:32:10.131054 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.130995    4931 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-244.us-west-2.compute.internal\" not found"
Nov 01 08:32:10.131261 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.131241    4931 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 08:32:10.131317 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.131308    4931 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 08:32:10.140434 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.140405    4931 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"]
Nov 01 08:32:10.140522 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.140505    4931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:10.141374 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.141357    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:10.141452 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.141383    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:10.141452 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.141397    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:10.141748 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.141735    4931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:10.141881 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.141857    4931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:10.142538 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142519    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:10.142610 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142548    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:10.142610 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142562    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:10.142610 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142577    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:10.142610 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142600    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:10.142732 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142620    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:10.142767 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.142746    4931 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-244.us-west-2.compute.internal\" not found" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.142815 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142792    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"36a0826bc10ea7285dbce782929bcb3b","Type":"ContainerStarted","Data":"a24937f6959662b79a550a9283edff0ad3011c0556d0bc5d96c5a9d18a63c24e"}
Nov 01 08:32:10.142844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142822    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"36a0826bc10ea7285dbce782929bcb3b","Type":"ContainerStarted","Data":"b8a9fac23e00f919297a254f0b8e41dbcca48a06164a7536e8eed8ac5d6396fe"}
Nov 01 08:32:10.142889 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142881    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"ca0fe841d453cc229ff1f7a2450a3ba4","Type":"ContainerStarted","Data":"6f74300c60dfb761454081416c6d00a9f30268ea873ea54ca20543daeb13e567"}
Nov 01 08:32:10.142922 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142895    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"ca0fe841d453cc229ff1f7a2450a3ba4","Type":"ContainerDied","Data":"22ff066c1e5e1aa34801ff452c3132f8487894816f1ecf19854f9943f6686135"}
Nov 01 08:32:10.142922 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142901    4931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:10.142922 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.142907    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal" event={"ID":"ca0fe841d453cc229ff1f7a2450a3ba4","Type":"ContainerStarted","Data":"3172d2f31dd0e27d233dc946cff3e5476f89e0650e43f02213088f214f22afe3"}
Nov 01 08:32:10.143705 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.143692    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:10.143768 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.143714    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:10.143768 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.143724    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:10.143839 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.143829    4931 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-244.us-west-2.compute.internal\" not found" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.187479 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.187432    4931 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:32:10.187479 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.187445    4931 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:32:10.188180 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.188161    4931 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 08:32:10.188318 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.188296    4931 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:32:10.209512 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.209493    4931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.229723 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.229704    4931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 08:32:10.233627 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.233611    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 08:32:10.233695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.233641    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 08:32:10.233695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.233657    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 08:32:10.233695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.233695    4931 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.236611 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.236593    4931 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 08:32:10.267840 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.267825    4931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:32:10.267926 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.267862    4931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.267926 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.267874    4931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.288713 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.288687    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36a0826bc10ea7285dbce782929bcb3b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"36a0826bc10ea7285dbce782929bcb3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.288770 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.288713    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.288770 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.288731    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.326342 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.326289    4931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:32:10.326342 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:10.326330    4931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.339280 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.339259    4931 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.339346 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.339319    4931 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.339842 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.339830    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeNotReady"
Nov 01 08:32:10.339901 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.339859    4931 setters.go:546] "Node became not ready" node="ip-10-0-143-244.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T08:32:10Z","lastTransitionTime":"2026-11-01T08:32:10Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 08:32:10.346173 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.346153    4931 nodeinfomanager.go:471] existing CSINode "ip-10-0-143-244.us-west-2.compute.internal" is owned by different node (oldNodeID="d3a88da2-88f5-449e-b536-fd96b19b09cc", newNodeID=""), cleaning up...
Nov 01 08:32:10.389370 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.389337    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36a0826bc10ea7285dbce782929bcb3b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"36a0826bc10ea7285dbce782929bcb3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.389370 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.389368    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.389544 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.389386    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.389544 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.389439    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.389544 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.389450    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36a0826bc10ea7285dbce782929bcb3b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"36a0826bc10ea7285dbce782929bcb3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.389544 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.389455    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca0fe841d453cc229ff1f7a2450a3ba4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" (UID: \"ca0fe841d453cc229ff1f7a2450a3ba4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:10.984841 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:10.984805    4931 apiserver.go:52] "Watching apiserver"
Nov 01 08:32:11.094676 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.094648    4931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:11.134988 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.134957    4931 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 08:32:11.136159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.136118    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427","openshift-ovn-kubernetes/ovnkube-node-bj2zg","openshift-console-operator/console-operator-7cd56b8dbc-jd869","openshift-image-registry/image-registry-78dd945d47-r7lx4","openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8","openshift-image-registry/node-ca-jzchj","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4","openshift-network-console/networking-console-plugin-86c478bb88-bbbls","openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk","openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm","openshift-multus/multus-additional-cni-plugins-jnc4r","openshift-insights/insights-operator-7c4986d449-62p6x","openshift-cluster-node-tuning-operator/tuned-cskq4","openshift-insights/insights-runtime-extractor-4g77m","openshift-multus/multus-2gdwv","kube-system/kas-connection-checker-685895c7c9-fg7b2","openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6","openshift-network-diagnostics/network-check-target-mnsws","openshift-dns/dns-default-8hhs6","openshift-image-registry/image-registry-6fd6466786-gpntg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh","kube-system/kas-connection-checker-685895c7c9-ftpbm","kube-system/konnectivity-agent-vlpgj","openshift-ingress-canary/ingress-canary-88hrp","kube-system/kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-nlwtz","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8","kube-system/kas-connection-checker-685895c7c9-67r69","openshift-dns/node-resolver-8t24r","openshift-ingress/router-default-c6bf599bf-9cjkf","kube-system/global-pull-secret-syncer-v7gkj","openshift-network-operator/iptables-alerter-ln7w5"]
Nov 01 08:32:11.139508 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.139489    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.139841 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.139826    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:11.140831 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.140807    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:11.141791 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.141774    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.142467 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.142451    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:11.147689 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.147671    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:11.149085 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.149066    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.150554 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.150538    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.151336 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.151322    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.155508 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.155479    4931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 08:32:11.182396 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.182354    4931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.us-west-2.compute.internal"
Nov 01 08:32:11.191808 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.191777    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.191978 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.191843    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 08:32:11.191978 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.191848    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.191978 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.191848    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 08:32:11.191978 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.191869    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.191978 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.191921    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.192295 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.192017    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 08:32:11.192503 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.192483    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.192758 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.192725    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.193377 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.193355    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.193449 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.193427    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.193720 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.193698    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.193850 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.193834    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.194015 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.193996    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 08:32:11.194097 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.194073    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.194226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.194210    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.194280 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.194254    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.195239 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.195218    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 08:32:11.195350 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.195284    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-r8vnl\""
Nov 01 08:32:11.195517 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.195499    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 08:32:11.195517 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.195513    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 08:32:11.195623 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.195553    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 08:32:11.195778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.195760    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 08:32:11.195835 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.195790    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 08:32:11.195950 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.195932    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-glvfr\""
Nov 01 08:32:11.196138 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.196120    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 08:32:11.212158 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.212139    4931 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 08:32:11.241131 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241059    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 08:32:11.241223 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241156    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.241223 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241188    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.241223 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241196    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 08:32:11.241223 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241197    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 08:32:11.241416 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241379    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.241416 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241388    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 08:32:11.241513 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241420    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.241513 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241469    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 08:32:11.241513 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241479    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.241513 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241492    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 08:32:11.241673 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241535    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 08:32:11.241673 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241547    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 08:32:11.241673 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241388    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 08:32:11.241673 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241598    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pn77d\""
Nov 01 08:32:11.241843 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241744    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 08:32:11.241843 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241761    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.241843 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241816    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 08:32:11.241843 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241827    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.242060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241870    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fph9z\""
Nov 01 08:32:11.242060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241876    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.242060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241876    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.242060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241909    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 08:32:11.242060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241993    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jftgd\""
Nov 01 08:32:11.242060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.241994    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.242271 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.242074    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-52479\""
Nov 01 08:32:11.242271 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.242085    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8lh4f\""
Nov 01 08:32:11.242271 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.242089    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 08:32:11.242271 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.242132    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ngh75\""
Nov 01 08:32:11.242271 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.242161    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 08:32:11.242475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.242338    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.242475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.242351    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 08:32:11.243174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.243161    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 08:32:11.249162 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249139    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.249244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249181    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.249244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249192    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 08:32:11.249244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249185    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 08:32:11.249244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249181    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.249468 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249439    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.249571 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249445    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 08:32:11.249571 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249525    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 08:32:11.249571 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249538    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.249571 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249552    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.249571 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249559    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249605    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249640    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249654    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249656    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249677    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249686    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249689    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249705    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249712    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249746    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249748    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 08:32:11.249812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.249757    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 08:32:11.291345 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291319    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 08:32:11.291499 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291373    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 08:32:11.291499 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291436    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxggb\""
Nov 01 08:32:11.291499 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291438    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hnxt\""
Nov 01 08:32:11.291617 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291546    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 08:32:11.291617 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291566    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mnx4s\""
Nov 01 08:32:11.291823 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291809    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 08:32:11.291869 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291821    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h4rfg\""
Nov 01 08:32:11.291869 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291813    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 08:32:11.291932 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.291882    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xx5dr\""
Nov 01 08:32:11.292215 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292197    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 08:32:11.292215 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292186    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-24cps\""
Nov 01 08:32:11.292340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292206    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xk5lv\""
Nov 01 08:32:11.292340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292225    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kmhw9\""
Nov 01 08:32:11.292340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292207    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-l9vpz\""
Nov 01 08:32:11.292340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292228    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jcrrb\""
Nov 01 08:32:11.292340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292291    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cdn26\""
Nov 01 08:32:11.292472 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292354    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48mlk\""
Nov 01 08:32:11.292472 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292408    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 08:32:11.292472 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292454    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 08:32:11.292589 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292531    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8d96l\""
Nov 01 08:32:11.292589 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292562    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8rfsj\""
Nov 01 08:32:11.292685 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292655    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xrtv\""
Nov 01 08:32:11.292725 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.292706    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqp75\""
Nov 01 08:32:11.295367 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295351    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cnibin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.295418 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295378    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cni-binary-copy\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.295418 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295397    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:11.295480 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295414    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxtr\" (UniqueName: \"kubernetes.io/projected/46b41145-f5ea-4af3-9f72-b12fee0dd5e2-kube-api-access-vgxtr\") pod \"kas-connection-checker-685895c7c9-fg7b2\" (UID: \"46b41145-f5ea-4af3-9f72-b12fee0dd5e2\") " pod="kube-system/kas-connection-checker-685895c7c9-fg7b2"
Nov 01 08:32:11.295515 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295484    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-var-lib-kubelet\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.295544 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295514    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-tuned\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.295665 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295639    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xkb\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.295665 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295649    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-tuned\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.295764 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295680    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08abdd05-c01c-4c69-ac14-330e24384e83-crio-socket\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.295815 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295796    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.295861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295829    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.295908 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295865    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-netd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.295954 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295905    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.296007 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.295953    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cni-binary-copy\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.296077 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296042    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-service-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.296156 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296143    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03c05080-9769-4bf4-ac9a-39b2c7301170-serving-cert\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:11.296209 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296188    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.296256 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296211    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.296256 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296221    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-netns\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.296256 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296245    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-service-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.296397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296261    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.296397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296303    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03c05080-9769-4bf4-ac9a-39b2c7301170-serving-cert\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:11.296397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296307    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.296397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296356    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-bin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.296563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296409    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-slash\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.296563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296446    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-snapshots\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.296563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296499    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-trusted-ca\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:11.296563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296508    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-snapshots\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.296563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296528    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.296563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296541    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:11.296870 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296639    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zfzs\" (UniqueName: \"kubernetes.io/projected/f1f77890-a449-4958-9b83-ef93acbe2f58-kube-api-access-7zfzs\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:11.296870 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296751    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.296870 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296780    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.296870 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296809    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-etc-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.296870 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296847    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296876    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e9c65da-47f6-494b-819e-34ee9896ff41-original-pull-secret\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296879    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-trusted-ca\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296902    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hdn\" (UniqueName: \"kubernetes.io/projected/de81ecfb-e731-40aa-9732-ca40772d9444-kube-api-access-67hdn\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296931    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89776b0e-b9bc-415b-8138-9ce3e9264a27-host-slash\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296945    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-system-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.296968    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-tmp\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297063    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297094    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5d1e97d-be9c-4f45-827c-19f789823a3b-konnectivity-ca\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297071    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a108cdc6-7b79-4e7a-8b7d-7824406258e3-tmp\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297097    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.297132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297129    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnngc\" (UniqueName: \"kubernetes.io/projected/2236c0f5-aab5-4979-8131-97477b58a186-kube-api-access-qnngc\") pod \"migrator-66df5d496b-sv7mk\" (UID: \"2236c0f5-aab5-4979-8131-97477b58a186\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297161    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297191    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526c4\" (UniqueName: \"kubernetes.io/projected/03c05080-9769-4bf4-ac9a-39b2c7301170-kube-api-access-526c4\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297224    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297230    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-k8s-cni-cncf-io\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297290    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5d1e97d-be9c-4f45-827c-19f789823a3b-konnectivity-ca\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297291    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-kubelet\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297377    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e38ced3e-d98c-4df1-935f-a1cdc2abf313-host\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297405    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-run\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297433    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297439    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:11.297506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297469    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.297794 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297537    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpkb\" (UniqueName: \"kubernetes.io/projected/6f5bef45-538f-465e-85cf-84a97fc9ba78-kube-api-access-ctpkb\") pod \"volume-data-source-validator-58c9f5f46d-75ht8\" (UID: \"6f5bef45-538f-465e-85cf-84a97fc9ba78\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"
Nov 01 08:32:11.297794 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297640    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.297794 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297687    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmk98\" (UniqueName: \"kubernetes.io/projected/608b447e-9dda-4503-bcef-26d6af784cfb-kube-api-access-hmk98\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:32:11.297873 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297845    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5czf\" (UniqueName: \"kubernetes.io/projected/89776b0e-b9bc-415b-8138-9ce3e9264a27-kube-api-access-q5czf\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:32:11.297902 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297882    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-config\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:11.297930 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297908    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-env-overrides\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.297970 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297935    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.298048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.297987    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-log-socket\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.298101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298069    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-bin\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.298136 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298106    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89q2k\" (UniqueName: \"kubernetes.io/projected/e38ced3e-d98c-4df1-935f-a1cdc2abf313-kube-api-access-89q2k\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:32:11.298226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298205    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-netns\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.298261 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298208    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-env-overrides\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.298294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298256    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pcz\" (UniqueName: \"kubernetes.io/projected/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-kube-api-access-66pcz\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.298294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298286    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-systemd-units\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.298392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298289    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03c05080-9769-4bf4-ac9a-39b2c7301170-config\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:11.298392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298316    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5kb\" (UniqueName: \"kubernetes.io/projected/afb159a0-e7ce-43e5-be5f-5260297660cb-kube-api-access-2k5kb\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:11.298392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298376    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qssqc\" (UniqueName: \"kubernetes.io/projected/ae17519f-2301-43ce-ac7e-890042574c55-kube-api-access-qssqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:32:11.298475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298417    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.298475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298454    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-kubelet-config\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:11.298530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298477    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-dbus\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:11.298530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298488    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.298586 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298565    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.298621 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298606    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:11.298658 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298636    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgd5t\" (UniqueName: \"kubernetes.io/projected/3ca1667a-32ed-4777-8755-ec7e72add722-kube-api-access-kgd5t\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.298695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298685    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae17519f-2301-43ce-ac7e-890042574c55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:32:11.298724 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298715    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-sys-fs\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.298761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298747    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.298802 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298762    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-conf-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.298832 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298797    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-node-log\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.298877 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298856    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.298905 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298875    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae17519f-2301-43ce-ac7e-890042574c55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:32:11.298905 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298875    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovn-node-metrics-cert\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.298964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298936    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-socket-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.299001 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.298985    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08abdd05-c01c-4c69-ac14-330e24384e83-data-volume\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.299072 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299058    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovn-node-metrics-cert\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.299110 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299068    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-script-lib\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.299110 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299092    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08abdd05-c01c-4c69-ac14-330e24384e83-data-volume\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.299179 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299127    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-ovn\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.299211 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299191    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.299239 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299217    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpw5p\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.299266 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299242    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:11.299316 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299300    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-script-lib\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.299349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299314    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/608b447e-9dda-4503-bcef-26d6af784cfb-hosts-file\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:32:11.299383 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299361    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9jm\" (UniqueName: \"kubernetes.io/projected/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-kube-api-access-qf9jm\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.299445 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299411    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.299494 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299463    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.299548 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299503    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1f77890-a449-4958-9b83-ef93acbe2f58-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:11.299548 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299531    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwtt\" (UniqueName: \"kubernetes.io/projected/c6f2d60e-838e-42df-8e42-58f036371709-kube-api-access-wzwtt\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.299638 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299554    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-kubelet\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.299638 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299579    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/608b447e-9dda-4503-bcef-26d6af784cfb-tmp-dir\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:32:11.299638 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299614    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299644    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05aad278-6386-4a54-89c9-c143151e86cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299668    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2d60e-838e-42df-8e42-58f036371709-tmp-dir\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299696    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-default-certificate\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299721    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-tmp\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299725    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/608b447e-9dda-4503-bcef-26d6af784cfb-tmp-dir\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299742    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1f77890-a449-4958-9b83-ef93acbe2f58-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299746    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gmq\" (UniqueName: \"kubernetes.io/projected/8800f430-6279-4e64-aef8-7c75e914b72b-kube-api-access-69gmq\") pod \"network-check-source-6f78d59f99-twnmm\" (UID: \"8800f430-6279-4e64-aef8-7c75e914b72b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299767    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f2d60e-838e-42df-8e42-58f036371709-tmp-dir\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299794    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.299817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299800    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ea361a-8d47-4aee-8ed8-b742628152ba-tmp\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299825    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299828    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05aad278-6386-4a54-89c9-c143151e86cf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299851    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-multus\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299879    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-sys\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299887    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299900    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-cert\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299916    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-serving-cert\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299926    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-default-certificate\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.299951    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300019    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-serving-cert\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300071    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/81ea361a-8d47-4aee-8ed8-b742628152ba-kube-api-access-wvttn\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300105    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5d1e97d-be9c-4f45-827c-19f789823a3b-agent-certs\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300119    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-cert\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300123    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-stats-auth\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300213    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-hostroot\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300243    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-lib-modules\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.300248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300259    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx899\" (UniqueName: \"kubernetes.io/projected/a108cdc6-7b79-4e7a-8b7d-7824406258e3-kube-api-access-bx899\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300288    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-stats-auth\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300328    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5d1e97d-be9c-4f45-827c-19f789823a3b-agent-certs\") pod \"konnectivity-agent-vlpgj\" (UID: \"e5d1e97d-be9c-4f45-827c-19f789823a3b\") " pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300395    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300431    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-config\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300478    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-etc-kubernetes\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300586    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-systemd\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300663    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-var-lib-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300690    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-ovnkube-config\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300695    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6997\" (UniqueName: \"kubernetes.io/projected/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-kube-api-access-s6997\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300733    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300763    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-socket-dir-parent\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300789    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysconfig\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300805    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300837    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-config\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300900    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.300976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300965    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-registration-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300991    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.300993    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-config\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301007    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301054    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/89776b0e-b9bc-415b-8138-9ce3e9264a27-iptables-alerter-script\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301142    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-kubernetes\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301205    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301237    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38ced3e-d98c-4df1-935f-a1cdc2abf313-serviceca\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301332    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/89776b0e-b9bc-415b-8138-9ce3e9264a27-iptables-alerter-script\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301338    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-host\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301382    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkqz\" (UniqueName: \"kubernetes.io/projected/bcd6ef98-6c6a-4687-bc85-4a540e06022a-kube-api-access-2fkqz\") pod \"kas-connection-checker-685895c7c9-ftpbm\" (UID: \"bcd6ef98-6c6a-4687-bc85-4a540e06022a\") " pod="kube-system/kas-connection-checker-685895c7c9-ftpbm"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301434    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-systemd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.301609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301458    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-device-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.302137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301853    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e38ced3e-d98c-4df1-935f-a1cdc2abf313-serviceca\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:32:11.302137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301915    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cnibin\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.302137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.301984    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae17519f-2301-43ce-ac7e-890042574c55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:32:11.302137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302042    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.302137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302128    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-multus-certs\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.302339 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302301    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f2d60e-838e-42df-8e42-58f036371709-config-volume\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.302388 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302344    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae17519f-2301-43ce-ac7e-890042574c55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:32:11.302433 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302383    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.302520 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302505    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-daemon-config\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.302575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302544    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.302575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302558    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ea361a-8d47-4aee-8ed8-b742628152ba-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.302638 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302589    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f2d60e-838e-42df-8e42-58f036371709-config-volume\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.302638 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302632    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt52q\" (UniqueName: \"kubernetes.io/projected/67b4458b-d683-447a-a253-5b37d8531843-kube-api-access-bt52q\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.302837 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302820    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.302896 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302833    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.302896 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302862    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81ea361a-8d47-4aee-8ed8-b742628152ba-serving-cert\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.302984 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302904    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-daemon-config\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.302984 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302947    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-modprobe-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.303103 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.302987    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-conf\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.303103 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.303052    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-522hk\" (UniqueName: \"kubernetes.io/projected/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-kube-api-access-522hk\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:32:11.303192 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.303126    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm7hr\" (UniqueName: \"kubernetes.io/projected/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-kube-api-access-xm7hr\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:32:11.303192 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.303169    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrt9t\" (UniqueName: \"kubernetes.io/projected/4f63fd1a-4af7-414d-aa39-4580e4e98d7f-kube-api-access-hrt9t\") pod \"kas-connection-checker-685895c7c9-67r69\" (UID: \"4f63fd1a-4af7-414d-aa39-4580e4e98d7f\") " pod="kube-system/kas-connection-checker-685895c7c9-67r69"
Nov 01 08:32:11.303929 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.303186    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81ea361a-8d47-4aee-8ed8-b742628152ba-serving-cert\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.304055 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.303988    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.304128 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.304085    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.304178 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.304127    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmds\" (UniqueName: \"kubernetes.io/projected/08abdd05-c01c-4c69-ac14-330e24384e83-kube-api-access-ndmds\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.304475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.304454    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.304541 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.304475    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.358303 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.358262    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxtr\" (UniqueName: \"kubernetes.io/projected/46b41145-f5ea-4af3-9f72-b12fee0dd5e2-kube-api-access-vgxtr\") pod \"kas-connection-checker-685895c7c9-fg7b2\" (UID: \"46b41145-f5ea-4af3-9f72-b12fee0dd5e2\") " pod="kube-system/kas-connection-checker-685895c7c9-fg7b2"
Nov 01 08:32:11.358514 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.358494    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.358613 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.358596    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zfzs\" (UniqueName: \"kubernetes.io/projected/f1f77890-a449-4958-9b83-ef93acbe2f58-kube-api-access-7zfzs\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:11.358768 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.358750    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xkb\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.361760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.361739    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnngc\" (UniqueName: \"kubernetes.io/projected/2236c0f5-aab5-4979-8131-97477b58a186-kube-api-access-qnngc\") pod \"migrator-66df5d496b-sv7mk\" (UID: \"2236c0f5-aab5-4979-8131-97477b58a186\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk"
Nov 01 08:32:11.361833 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.361813    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pcz\" (UniqueName: \"kubernetes.io/projected/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-kube-api-access-66pcz\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.361867 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.361838    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89q2k\" (UniqueName: \"kubernetes.io/projected/e38ced3e-d98c-4df1-935f-a1cdc2abf313-kube-api-access-89q2k\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:32:11.361897 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.361875    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx899\" (UniqueName: \"kubernetes.io/projected/a108cdc6-7b79-4e7a-8b7d-7824406258e3-kube-api-access-bx899\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.363490 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.363464    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpkb\" (UniqueName: \"kubernetes.io/projected/6f5bef45-538f-465e-85cf-84a97fc9ba78-kube-api-access-ctpkb\") pod \"volume-data-source-validator-58c9f5f46d-75ht8\" (UID: \"6f5bef45-538f-465e-85cf-84a97fc9ba78\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-75ht8"
Nov 01 08:32:11.363676 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.363660    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9jm\" (UniqueName: \"kubernetes.io/projected/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-kube-api-access-qf9jm\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.363676 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.363667    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmds\" (UniqueName: \"kubernetes.io/projected/08abdd05-c01c-4c69-ac14-330e24384e83-kube-api-access-ndmds\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.363830 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.363817    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qpf\" (UniqueName: \"kubernetes.io/projected/44a34856-dd40-4408-9db2-67d075d9850a-kube-api-access-q6qpf\") pod \"network-check-target-mnsws\" (UID: \"44a34856-dd40-4408-9db2-67d075d9850a\") " pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:11.363878 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.363836    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qssqc\" (UniqueName: \"kubernetes.io/projected/ae17519f-2301-43ce-ac7e-890042574c55-kube-api-access-qssqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4\" (UID: \"ae17519f-2301-43ce-ac7e-890042574c55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4"
Nov 01 08:32:11.363965 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.363949    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt52q\" (UniqueName: \"kubernetes.io/projected/67b4458b-d683-447a-a253-5b37d8531843-kube-api-access-bt52q\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.366478 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.366459    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-522hk\" (UniqueName: \"kubernetes.io/projected/29a037fa-abd6-4bc2-9bc1-0d7da8429b09-kube-api-access-522hk\") pod \"ingress-canary-88hrp\" (UID: \"29a037fa-abd6-4bc2-9bc1-0d7da8429b09\") " pod="openshift-ingress-canary/ingress-canary-88hrp"
Nov 01 08:32:11.366544 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.366477    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hdn\" (UniqueName: \"kubernetes.io/projected/de81ecfb-e731-40aa-9732-ca40772d9444-kube-api-access-67hdn\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:11.366578 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.366549    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwtt\" (UniqueName: \"kubernetes.io/projected/c6f2d60e-838e-42df-8e42-58f036371709-kube-api-access-wzwtt\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.366795 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.366779    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkqz\" (UniqueName: \"kubernetes.io/projected/bcd6ef98-6c6a-4687-bc85-4a540e06022a-kube-api-access-2fkqz\") pod \"kas-connection-checker-685895c7c9-ftpbm\" (UID: \"bcd6ef98-6c6a-4687-bc85-4a540e06022a\") " pod="kube-system/kas-connection-checker-685895c7c9-ftpbm"
Nov 01 08:32:11.366943 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.366928    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgd5t\" (UniqueName: \"kubernetes.io/projected/3ca1667a-32ed-4777-8755-ec7e72add722-kube-api-access-kgd5t\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.369359 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.369342    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5czf\" (UniqueName: \"kubernetes.io/projected/89776b0e-b9bc-415b-8138-9ce3e9264a27-kube-api-access-q5czf\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:32:11.371526 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.371504    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5kb\" (UniqueName: \"kubernetes.io/projected/afb159a0-e7ce-43e5-be5f-5260297660cb-kube-api-access-2k5kb\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:11.371656 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.371640    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpw5p\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.371727 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.371662    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrt9t\" (UniqueName: \"kubernetes.io/projected/4f63fd1a-4af7-414d-aa39-4580e4e98d7f-kube-api-access-hrt9t\") pod \"kas-connection-checker-685895c7c9-67r69\" (UID: \"4f63fd1a-4af7-414d-aa39-4580e4e98d7f\") " pod="kube-system/kas-connection-checker-685895c7c9-67r69"
Nov 01 08:32:11.371727 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.371694    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526c4\" (UniqueName: \"kubernetes.io/projected/03c05080-9769-4bf4-ac9a-39b2c7301170-kube-api-access-526c4\") pod \"console-operator-7cd56b8dbc-jd869\" (UID: \"03c05080-9769-4bf4-ac9a-39b2c7301170\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:11.372377 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.372359    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gmq\" (UniqueName: \"kubernetes.io/projected/8800f430-6279-4e64-aef8-7c75e914b72b-kube-api-access-69gmq\") pod \"network-check-source-6f78d59f99-twnmm\" (UID: \"8800f430-6279-4e64-aef8-7c75e914b72b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-twnmm"
Nov 01 08:32:11.372525 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.372508    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6997\" (UniqueName: \"kubernetes.io/projected/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-kube-api-access-s6997\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.373192 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.373177    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/81ea361a-8d47-4aee-8ed8-b742628152ba-kube-api-access-wvttn\") pod \"insights-operator-7c4986d449-62p6x\" (UID: \"81ea361a-8d47-4aee-8ed8-b742628152ba\") " pod="openshift-insights/insights-operator-7c4986d449-62p6x"
Nov 01 08:32:11.376656 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.376640    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm7hr\" (UniqueName: \"kubernetes.io/projected/2023ca6f-9936-4e4c-80a1-e33b47fa7dcb-kube-api-access-xm7hr\") pod \"service-ca-operator-544d749c8c-9lnx8\" (UID: \"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8"
Nov 01 08:32:11.377345 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.377330    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmk98\" (UniqueName: \"kubernetes.io/projected/608b447e-9dda-4503-bcef-26d6af784cfb-kube-api-access-hmk98\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:32:11.379134 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.379113    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.404893 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.404850    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-sys\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.404902    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-hostroot\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.404925    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-lib-modules\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.404960    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-etc-kubernetes\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.404969    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-sys\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.404985    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-systemd\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.404970    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-hostroot\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405003    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-etc-kubernetes\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405016    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-var-lib-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405056    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-systemd\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405062    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405083    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-lib-modules\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405089    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-socket-dir-parent\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405096    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-var-lib-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405114    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysconfig\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405140    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405143    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-socket-dir-parent\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405164    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405182    4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405202    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysconfig\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405209    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-registration-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405183    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-registration-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405251    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.90523472 +0000 UTC m=+2.020107688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405238    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405276    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405293    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405324    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-kubernetes\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405339    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405355    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-host\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405358    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-kubernetes\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405374    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-systemd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405386    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405393    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-device-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405408    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cnibin\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405412    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-systemd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405414    4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405416    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-host\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405424    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405443    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-device-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405464    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.905447887 +0000 UTC m=+2.020320833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405474    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405479    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-multus-certs\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405476    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-cnibin\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405499    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-multus-certs\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405510    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405538    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405554    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-modprobe-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405569    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-conf\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405596    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cnibin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405619    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405625    4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405641    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-var-lib-kubelet\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405664    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.905651702 +0000 UTC m=+2.020524678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405681    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-modprobe-d\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405691    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-var-lib-kubelet\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405700    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.905689986 +0000 UTC m=+2.020562938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405701    4931 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405710    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-etc-sysctl-conf\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405686    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-cnibin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.405750    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.905738961 +0000 UTC m=+2.020611920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : secret "metrics-daemon-secret" not found
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405683    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08abdd05-c01c-4c69-ac14-330e24384e83-crio-socket\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405765    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08abdd05-c01c-4c69-ac14-330e24384e83-crio-socket\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405775    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405793    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-netd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405817    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-netns\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405817    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-netd\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405840    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405859    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405864    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-run-netns\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405873    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-bin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405901    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-slash\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405925    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-slash\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405904    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-bin\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405933    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405940    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405940    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-etc-selinux\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.405971    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-etc-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.405989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406043    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89776b0e-b9bc-415b-8138-9ce3e9264a27-host-slash\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406056    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406066    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-system-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406089    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89776b0e-b9bc-415b-8138-9ce3e9264a27-host-slash\") pod \"iptables-alerter-ln7w5\" (UID: \"89776b0e-b9bc-415b-8138-9ce3e9264a27\") " pod="openshift-network-operator/iptables-alerter-ln7w5"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406103    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-etc-openvswitch\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406114    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-k8s-cni-cncf-io\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406134    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-k8s-cni-cncf-io\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406138    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-kubelet\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406160    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e38ced3e-d98c-4df1-935f-a1cdc2abf313-host\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406115    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-system-cni-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406179    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-kubelet\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406188    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-run\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406201    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e38ced3e-d98c-4df1-935f-a1cdc2abf313-host\") pod \"node-ca-jzchj\" (UID: \"e38ced3e-d98c-4df1-935f-a1cdc2abf313\") " pod="openshift-image-registry/node-ca-jzchj"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406209    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406231    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-log-socket\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406246    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-bin\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406247    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a108cdc6-7b79-4e7a-8b7d-7824406258e3-run\") pod \"tuned-cskq4\" (UID: \"a108cdc6-7b79-4e7a-8b7d-7824406258e3\") " pod="openshift-cluster-node-tuning-operator/tuned-cskq4"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406262    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-netns\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406276    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-cni-bin\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406295    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-systemd-units\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406300    4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406315    4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd6466786-gpntg: secret "image-registry-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406286    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-log-socket\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406322    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-run-netns\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406278    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-systemd-units\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406326    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08abdd05-c01c-4c69-ac14-330e24384e83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406397    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls podName:6ddfd7c8-e096-40dc-8e3d-a359cb566176 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.906385371 +0000 UTC m=+2.021258334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls") pod "image-registry-6fd6466786-gpntg" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176") : secret "image-registry-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406412    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-kubelet-config\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406427    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-dbus\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406449    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406475    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-sys-fs\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406479    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-kubelet-config\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406496    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e9c65da-47f6-494b-819e-34ee9896ff41-dbus\") pod \"global-pull-secret-syncer-v7gkj\" (UID: \"2e9c65da-47f6-494b-819e-34ee9896ff41\") " pod="kube-system/global-pull-secret-syncer-v7gkj"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406517    4931 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406519    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-sys-fs\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406534    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-conf-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406547    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert podName:05aad278-6386-4a54-89c9-c143151e86cf nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.906536625 +0000 UTC m=+2.021409603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bbbls" (UID: "05aad278-6386-4a54-89c9-c143151e86cf") : secret "networking-console-plugin-cert" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406518    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-multus-conf-dir\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406570    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-node-log\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406586    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-socket-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406612    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-node-log\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406626    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67b4458b-d683-447a-a253-5b37d8531843-socket-dir\") pod \"aws-ebs-csi-driver-node-fqfsh\" (UID: \"67b4458b-d683-447a-a253-5b37d8531843\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fqfsh"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406630    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-ovn\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406646    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-run-ovn\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406657    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406674    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406693    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/608b447e-9dda-4503-bcef-26d6af784cfb-hosts-file\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406704    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a214d76-85af-4fd5-9ccd-6cdce46fdd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnc4r\" (UID: \"3a214d76-85af-4fd5-9ccd-6cdce46fdd6e\") " pod="openshift-multus/multus-additional-cni-plugins-jnc4r"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406710    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406733    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406739    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/608b447e-9dda-4503-bcef-26d6af784cfb-hosts-file\") pod \"node-resolver-8t24r\" (UID: \"608b447e-9dda-4503-bcef-26d6af784cfb\") " pod="openshift-dns/node-resolver-8t24r"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406765    4931 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406766    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f13e4499-bb2f-4e37-bd37-52f4bdad02dd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bj2zg\" (UID: \"f13e4499-bb2f-4e37-bd37-52f4bdad02dd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406803    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.906792791 +0000 UTC m=+2.021665736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406809    4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406822    4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dd945d47-r7lx4: secret "image-registry-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406834    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-kubelet\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406861    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-kubelet\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406865    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406882    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls podName:5974cc8f-4fe4-4143-bd79-d1b8888fc1ab nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.906867298 +0000 UTC m=+2.021740267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls") pod "image-registry-78dd945d47-r7lx4" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab") : secret "image-registry-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406910    4931 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406921    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-multus\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.406940    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:11.906930973 +0000 UTC m=+2.021803936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:11.407797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.406993    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900-host-var-lib-cni-multus\") pod \"multus-2gdwv\" (UID: \"8f8bc7e3-ca5d-4db2-8a63-9b48e38b5900\") " pod="openshift-multus/multus-2gdwv"
Nov 01 08:32:11.440722 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.440700    4931 scope.go:122] "RemoveContainer" containerID="92603f6a7e867a98d077b9a5c80c979a19f2d98a80e4b0aff735d8291e6bd7e0"
Nov 01 08:32:11.453849 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.453830    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:32:11.892565 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.892539    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:11.911831 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.911806    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:11.911984 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.911853    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:11.911984 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.911963    4931 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:11.912109 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.911985    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:11.912109 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.911967    4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:11.912109 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912051    4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd6466786-gpntg: secret "image-registry-tls" not found
Nov 01 08:32:11.912109 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.912054    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:11.912109 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912090    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls podName:6ddfd7c8-e096-40dc-8e3d-a359cb566176 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.91207485 +0000 UTC m=+3.026947809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls") pod "image-registry-6fd6466786-gpntg" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176") : secret "image-registry-tls" not found
Nov 01 08:32:11.912109 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912110    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.912104405 +0000 UTC m=+3.026977350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : secret "metrics-daemon-secret" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912126    4931 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912134    4931 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912175    4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912174    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.91216274 +0000 UTC m=+3.027035685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.912130    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912195    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert podName:05aad278-6386-4a54-89c9-c143151e86cf nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.912188779 +0000 UTC m=+3.027061724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bbbls" (UID: "05aad278-6386-4a54-89c9-c143151e86cf") : secret "networking-console-plugin-cert" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912183    4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dd945d47-r7lx4: secret "image-registry-tls" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912253    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls podName:5974cc8f-4fe4-4143-bd79-d1b8888fc1ab nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.912244556 +0000 UTC m=+3.027117502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls") pod "image-registry-78dd945d47-r7lx4" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab") : secret "image-registry-tls" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.912252    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.912288    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912292    4931 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.912313    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912323    4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912353    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.912338413 +0000 UTC m=+3.027211372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:11.912361 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912374    4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:11.912917 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.912339    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.912917 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912375    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.91236594 +0000 UTC m=+3.027238890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:11.912917 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912401    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.912394023 +0000 UTC m=+3.027266968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:11.912917 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912434    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.912414007 +0000 UTC m=+3.027286956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:32:11.912917 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:11.912449    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:11.912917 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912520    4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:32:11.912917 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:11.912550    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:12.91254309 +0000 UTC m=+3.027416048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:32:12.098968 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.098944    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:32:12.099416 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.099397    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/1.log"
Nov 01 08:32:12.099456 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.099426    4931 generic.go:356] "Generic (PLEG): container finished" podID="03c05080-9769-4bf4-ac9a-39b2c7301170" containerID="28e8faf717cdc153879ebc120ec4335e083893115109197639629d9db21d510c" exitCode=255
Nov 01 08:32:12.099526 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.099503    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" event={"ID":"03c05080-9769-4bf4-ac9a-39b2c7301170","Type":"ContainerDied","Data":"28e8faf717cdc153879ebc120ec4335e083893115109197639629d9db21d510c"}
Nov 01 08:32:12.099599 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.099584    4931 scope.go:122] "RemoveContainer" containerID="92603f6a7e867a98d077b9a5c80c979a19f2d98a80e4b0aff735d8291e6bd7e0"
Nov 01 08:32:12.100208 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.100184    4931 scope.go:122] "RemoveContainer" containerID="28e8faf717cdc153879ebc120ec4335e083893115109197639629d9db21d510c"
Nov 01 08:32:12.100383 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.100366    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jd869_openshift-console-operator(03c05080-9769-4bf4-ac9a-39b2c7301170)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" podUID="03c05080-9769-4bf4-ac9a-39b2c7301170"
Nov 01 08:32:12.922310 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922278    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:12.922310 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922315    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922341    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922415    4931 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922419    4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922437    4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fd6466786-gpntg: secret "image-registry-tls" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922457    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.92244531 +0000 UTC m=+5.037318257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922453    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922417    4931 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922484    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls podName:6ddfd7c8-e096-40dc-8e3d-a359cb566176 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.922471402 +0000 UTC m=+5.037344363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls") pod "image-registry-6fd6466786-gpntg" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176") : secret "image-registry-tls" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922501    4931 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922506    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert podName:05aad278-6386-4a54-89c9-c143151e86cf nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.922495527 +0000 UTC m=+5.037368473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bbbls" (UID: "05aad278-6386-4a54-89c9-c143151e86cf") : secret "networking-console-plugin-cert" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922509    4931 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dd945d47-r7lx4: secret "image-registry-tls" not found
Nov 01 08:32:12.922530 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922530    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls podName:5974cc8f-4fe4-4143-bd79-d1b8888fc1ab nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.922524203 +0000 UTC m=+5.037397148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls") pod "image-registry-78dd945d47-r7lx4" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab") : secret "image-registry-tls" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922544    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922575    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922603    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922627    4931 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922639    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922667    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922681    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.922665482 +0000 UTC m=+5.037538447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922683    4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922713    4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922721    4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:12.922725    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922730    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.922716175 +0000 UTC m=+5.037589134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922757    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.922746065 +0000 UTC m=+5.037619111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922768    4931 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922776    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.92276691 +0000 UTC m=+5.037639858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922807    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.922787531 +0000 UTC m=+5.037660490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:32:12.922989 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:12.922854    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs podName:afb159a0-e7ce-43e5-be5f-5260297660cb nodeName:}" failed. No retries permitted until 2026-11-01 08:32:14.92284623 +0000 UTC m=+5.037719175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs") pod "network-metrics-daemon-nlwtz" (UID: "afb159a0-e7ce-43e5-be5f-5260297660cb") : secret "metrics-daemon-secret" not found
Nov 01 08:32:13.074802 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:13.074774    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:13.103753 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:13.103724    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sv7mk" event={"ID":"2236c0f5-aab5-4979-8131-97477b58a186","Type":"ContainerStarted","Data":"36083b9d306aaffc4713d3e49f2c8a4e7c4e6e1e70fc038106ede1a62f1bbdcd"}
Nov 01 08:32:13.105285 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:13.105265    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:32:13.105450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:13.105436    4931 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 08:32:14.942749 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.942707    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:14.942749 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.942756    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.942785    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.942820    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.942847    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.942862    4931 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.942875    4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.942880    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.942937    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:18.942918895 +0000 UTC m=+9.057791859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.942959    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:18.942950235 +0000 UTC m=+9.057823180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.942976    4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.942979    4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.943006    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:18.942986431 +0000 UTC m=+9.057859390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.943068    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:18.943051593 +0000 UTC m=+9.057924545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.943093    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:18.94308658 +0000 UTC m=+9.057959525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.943131    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.943159    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.943187    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.943215    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.943230    4931 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.943271    4931 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 08:32:14.943294 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:14.943312    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls podName:de81ecfb-e731-40aa-9732-ca40772d9444 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:18.943299738 +0000 UTC m=+9.058172697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mf427" (UID: "de81ecfb-e731-40aa-9732-ca40772d9444") : secret "samples-operator-tls" not found
Nov 01 08:32:14.945515 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.945497    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb159a0-e7ce-43e5-be5f-5260297660cb-metrics-certs\") pod \"network-metrics-daemon-nlwtz\" (UID: \"afb159a0-e7ce-43e5-be5f-5260297660cb\") " pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:14.945613 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.945561    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"image-registry-6fd6466786-gpntg\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") " pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:14.959262 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.959240    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"image-registry-78dd945d47-r7lx4\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") " pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:14.959360 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:14.959278    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05aad278-6386-4a54-89c9-c143151e86cf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bbbls\" (UID: \"05aad278-6386-4a54-89c9-c143151e86cf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:15.042427 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.042401    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:15.043483 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.043464    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls"
Nov 01 08:32:15.048608 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.048590    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlwtz"
Nov 01 08:32:15.049698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.049684    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:15.391206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.391184    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8t24r_608b447e-9dda-4503-bcef-26d6af784cfb/dns-node-resolver/0.log"
Nov 01 08:32:15.518616 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.518455    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bbbls"]
Nov 01 08:32:15.522005 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:15.521955    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aad278_6386_4a54_89c9_c143151e86cf.slice/crio-7ed14a3c803ff6d8730e01a8129f2b15a91458a09d3283a76a8978c6b21430f6 WatchSource:0}: Error finding container 7ed14a3c803ff6d8730e01a8129f2b15a91458a09d3283a76a8978c6b21430f6: Status 404 returned error can't find the container with id 7ed14a3c803ff6d8730e01a8129f2b15a91458a09d3283a76a8978c6b21430f6
Nov 01 08:32:15.522809 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.522681    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78dd945d47-r7lx4"]
Nov 01 08:32:15.525068 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:15.525046    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5974cc8f_4fe4_4143_bd79_d1b8888fc1ab.slice/crio-09c8697b50e517d3b2ff8b31b19482bb0c405f77dce2b365ae5412d108181fd6 WatchSource:0}: Error finding container 09c8697b50e517d3b2ff8b31b19482bb0c405f77dce2b365ae5412d108181fd6: Status 404 returned error can't find the container with id 09c8697b50e517d3b2ff8b31b19482bb0c405f77dce2b365ae5412d108181fd6
Nov 01 08:32:15.541060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.541013    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fd6466786-gpntg"]
Nov 01 08:32:15.564462 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:15.564440    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nlwtz"]
Nov 01 08:32:15.567758 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:15.567733    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddfd7c8_e096_40dc_8e3d_a359cb566176.slice/crio-f401ca95eb5d256f6b3064342e2d37fd627fbb4a796e082831d15c52e2ec0164 WatchSource:0}: Error finding container f401ca95eb5d256f6b3064342e2d37fd627fbb4a796e082831d15c52e2ec0164: Status 404 returned error can't find the container with id f401ca95eb5d256f6b3064342e2d37fd627fbb4a796e082831d15c52e2ec0164
Nov 01 08:32:15.568409 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:15.568389    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb159a0_e7ce_43e5_be5f_5260297660cb.slice/crio-c7cd1d6525edd76cf8b27fb46248091ee6b20d3e7c8fe82b274ad408f5cca584 WatchSource:0}: Error finding container c7cd1d6525edd76cf8b27fb46248091ee6b20d3e7c8fe82b274ad408f5cca584: Status 404 returned error can't find the container with id c7cd1d6525edd76cf8b27fb46248091ee6b20d3e7c8fe82b274ad408f5cca584
Nov 01 08:32:16.127382 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:16.127343    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" event={"ID":"6ddfd7c8-e096-40dc-8e3d-a359cb566176","Type":"ContainerStarted","Data":"922f008f848046a955b3d4d188a870c8227e882810b2e86a20c83b7a13d55e78"}
Nov 01 08:32:16.127862 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:16.127442    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" event={"ID":"6ddfd7c8-e096-40dc-8e3d-a359cb566176","Type":"ContainerStarted","Data":"f401ca95eb5d256f6b3064342e2d37fd627fbb4a796e082831d15c52e2ec0164"}
Nov 01 08:32:16.128844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:16.128821    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlwtz" event={"ID":"afb159a0-e7ce-43e5-be5f-5260297660cb","Type":"ContainerStarted","Data":"c7cd1d6525edd76cf8b27fb46248091ee6b20d3e7c8fe82b274ad408f5cca584"}
Nov 01 08:32:16.129931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:16.129912    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls" event={"ID":"05aad278-6386-4a54-89c9-c143151e86cf","Type":"ContainerStarted","Data":"7ed14a3c803ff6d8730e01a8129f2b15a91458a09d3283a76a8978c6b21430f6"}
Nov 01 08:32:16.131393 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:16.131376    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4" event={"ID":"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab","Type":"ContainerStarted","Data":"794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b"}
Nov 01 08:32:16.131447 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:16.131410    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4" event={"ID":"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab","Type":"ContainerStarted","Data":"09c8697b50e517d3b2ff8b31b19482bb0c405f77dce2b365ae5412d108181fd6"}
Nov 01 08:32:16.188707 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:16.188680    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jzchj_e38ced3e-d98c-4df1-935f-a1cdc2abf313/node-ca/0.log"
Nov 01 08:32:17.135463 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:17.135431    4931 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 08:32:17.135949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:17.135484    4931 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 08:32:17.191166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:17.191145    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-88hrp_29a037fa-abd6-4bc2-9bc1-0d7da8429b09/serve-healthcheck-canary/0.log"
Nov 01 08:32:17.249978 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:17.249952    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:17.250450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:17.250428    4931 scope.go:122] "RemoveContainer" containerID="28e8faf717cdc153879ebc120ec4335e083893115109197639629d9db21d510c"
Nov 01 08:32:17.250686 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:17.250647    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jd869_openshift-console-operator(03c05080-9769-4bf4-ac9a-39b2c7301170)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" podUID="03c05080-9769-4bf4-ac9a-39b2c7301170"
Nov 01 08:32:17.589003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:17.588929    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-sv7mk_2236c0f5-aab5-4979-8131-97477b58a186/migrator/0.log"
Nov 01 08:32:17.790405 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:17.790368    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4_ae17519f-2301-43ce-ac7e-890042574c55/kube-storage-version-migrator-operator/0.log"
Nov 01 08:32:18.140685 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.140651    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlwtz" event={"ID":"afb159a0-e7ce-43e5-be5f-5260297660cb","Type":"ContainerStarted","Data":"ca0134abb8367db103806000cd0b181d9a022937f779cf9c676d93368e79b4f1"}
Nov 01 08:32:18.141199 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.140712    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlwtz" event={"ID":"afb159a0-e7ce-43e5-be5f-5260297660cb","Type":"ContainerStarted","Data":"b1e489b15ab476684df71b5af83870355b4863f726fe3f3ad1234f3a72f11b52"}
Nov 01 08:32:18.142149 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.142126    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-bbbls" event={"ID":"05aad278-6386-4a54-89c9-c143151e86cf","Type":"ContainerStarted","Data":"88fcdb1929cc2d4f156fbece54d2411886ee2ea48381d0d819721da21ee6f0ca"}
Nov 01 08:32:18.382276 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.382245    4931 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:18.382586 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.382572    4931 scope.go:122] "RemoveContainer" containerID="28e8faf717cdc153879ebc120ec4335e083893115109197639629d9db21d510c"
Nov 01 08:32:18.382753 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.382737    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jd869_openshift-console-operator(03c05080-9769-4bf4-ac9a-39b2c7301170)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" podUID="03c05080-9769-4bf4-ac9a-39b2c7301170"
Nov 01 08:32:18.386701 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.386683    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:18.982743 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.982714    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.982753    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.982777    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.982796    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.982817    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.982834    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.982867    4931 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.982876    4931 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.982929    4931 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 08:32:18.982943 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.982935    4931 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 08:32:18.983357 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.983110    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls podName:f1f77890-a449-4958-9b83-ef93acbe2f58 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:26.982920908 +0000 UTC m=+17.097793873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bksj6" (UID: "f1f77890-a449-4958-9b83-ef93acbe2f58") : secret "cluster-monitoring-operator-tls" not found
Nov 01 08:32:18.983453 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.983378    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls podName:08abdd05-c01c-4c69-ac14-330e24384e83 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:26.983359655 +0000 UTC m=+17.098232600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4g77m" (UID: "08abdd05-c01c-4c69-ac14-330e24384e83") : secret "insights-runtime-extractor-tls" not found
Nov 01 08:32:18.983453 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.983396    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:26.983386518 +0000 UTC m=+17.098259470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : configmap references non-existent config key: service-ca.crt
Nov 01 08:32:18.983453 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.983425    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls podName:c6f2d60e-838e-42df-8e42-58f036371709 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:26.983403173 +0000 UTC m=+17.098276120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls") pod "dns-default-8hhs6" (UID: "c6f2d60e-838e-42df-8e42-58f036371709") : secret "dns-default-metrics-tls" not found
Nov 01 08:32:18.983453 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:18.983440    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs podName:3ca1667a-32ed-4777-8755-ec7e72add722 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:26.983432047 +0000 UTC m=+17.098305007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs") pod "router-default-c6bf599bf-9cjkf" (UID: "3ca1667a-32ed-4777-8755-ec7e72add722") : secret "router-metrics-certs-default" not found
Nov 01 08:32:18.989305 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:18.989277    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de81ecfb-e731-40aa-9732-ca40772d9444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mf427\" (UID: \"de81ecfb-e731-40aa-9732-ca40772d9444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:19.241924 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:19.241846    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"
Nov 01 08:32:19.263776 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:19.263752    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:19.602207 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:19.602184    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427"]
Nov 01 08:32:20.149818 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:20.149787    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427" event={"ID":"de81ecfb-e731-40aa-9732-ca40772d9444","Type":"ContainerStarted","Data":"65ca1660895b5bedace52758ea1b276bdd58ea3f68e0ff914a29c6c8ba9a82a6"}
Nov 01 08:32:20.253356 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:20.253321    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:20.404572 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:20.404501    4931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.us-west-2.compute.internal" event="NodeReady"
Nov 01 08:32:20.723837 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:20.723764    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vlpgj"
Nov 01 08:32:22.041459 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:22.041426    4931 scope.go:122] "RemoveContainer" containerID="28e8faf717cdc153879ebc120ec4335e083893115109197639629d9db21d510c"
Nov 01 08:32:22.160305 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:22.160269    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427" event={"ID":"de81ecfb-e731-40aa-9732-ca40772d9444","Type":"ContainerStarted","Data":"f44cf35dce4eca4c2acf5dede6cad045767f60b03708dccdf10e883c37abb1b0"}
Nov 01 08:32:22.160479 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:22.160331    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mf427" event={"ID":"de81ecfb-e731-40aa-9732-ca40772d9444","Type":"ContainerStarted","Data":"fe1ab76ec9f7eb75cc69e40463784f01ce19b77f1dd4f5655df5e6ca152501d0"}
Nov 01 08:32:22.162151 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:22.162132    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:32:22.162246 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:22.162195    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" event={"ID":"03c05080-9769-4bf4-ac9a-39b2c7301170","Type":"ContainerStarted","Data":"5789247f9d9ddbf72277831626d3ab814fcc41d8784d70d98a04189af6bd1de3"}
Nov 01 08:32:22.162501 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:22.162473    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:22.163727 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:22.163705    4931 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jd869 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.10:8443/readyz\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body=
Nov 01 08:32:22.163783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:22.163754    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869" podUID="03c05080-9769-4bf4-ac9a-39b2c7301170" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.10:8443/readyz\": dial tcp 10.133.0.10:8443: connect: connection refused"
Nov 01 08:32:23.169501 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:23.169474    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jd869"
Nov 01 08:32:27.058317 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.058278    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:27.058317 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.058324    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:27.058793 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.058349    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:27.058793 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.058371    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:27.058793 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.058400    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:27.059136 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.059110    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca1667a-32ed-4777-8755-ec7e72add722-service-ca-bundle\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:27.060988 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.060968    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1f77890-a449-4958-9b83-ef93acbe2f58-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bksj6\" (UID: \"f1f77890-a449-4958-9b83-ef93acbe2f58\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:27.061100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.061067    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca1667a-32ed-4777-8755-ec7e72add722-metrics-certs\") pod \"router-default-c6bf599bf-9cjkf\" (UID: \"3ca1667a-32ed-4777-8755-ec7e72add722\") " pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:27.061333 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.061313    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6f2d60e-838e-42df-8e42-58f036371709-metrics-tls\") pod \"dns-default-8hhs6\" (UID: \"c6f2d60e-838e-42df-8e42-58f036371709\") " pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:27.061401 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.061360    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08abdd05-c01c-4c69-ac14-330e24384e83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4g77m\" (UID: \"08abdd05-c01c-4c69-ac14-330e24384e83\") " pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:27.339909 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.339819    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:27.340873 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.340854    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"
Nov 01 08:32:27.351236 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.351191    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:27.356835 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.352976    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4g77m"
Nov 01 08:32:27.410747 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.410713    4931 patch_prober.go:29] interesting pod/image-registry-78dd945d47-r7lx4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 08:32:27.410885 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.410784    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4" podUID="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 08:32:27.735699 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.735669    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6"]
Nov 01 08:32:27.751133 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:27.751103    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f77890_a449_4958_9b83_ef93acbe2f58.slice/crio-aecd15dc0fc12380b56d709d2db9e61c2dc409a33e2eef9165cbcbd12132d7fc WatchSource:0}: Error finding container aecd15dc0fc12380b56d709d2db9e61c2dc409a33e2eef9165cbcbd12132d7fc: Status 404 returned error can't find the container with id aecd15dc0fc12380b56d709d2db9e61c2dc409a33e2eef9165cbcbd12132d7fc
Nov 01 08:32:27.764073 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:27.764044    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08abdd05_c01c_4c69_ac14_330e24384e83.slice/crio-3f8bc9f3a791d01b80b0d6597e8a1ed3d59df6318feb8863f5f91ba354f9a93b WatchSource:0}: Error finding container 3f8bc9f3a791d01b80b0d6597e8a1ed3d59df6318feb8863f5f91ba354f9a93b: Status 404 returned error can't find the container with id 3f8bc9f3a791d01b80b0d6597e8a1ed3d59df6318feb8863f5f91ba354f9a93b
Nov 01 08:32:27.783377 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.783354    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4g77m"]
Nov 01 08:32:27.784005 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.783988    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c6bf599bf-9cjkf"]
Nov 01 08:32:27.797463 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:27.797435    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca1667a_32ed_4777_8755_ec7e72add722.slice/crio-8aef130603fa74ae92cffd87e9407f208e7fac285d49964703f47d97ad95dad5 WatchSource:0}: Error finding container 8aef130603fa74ae92cffd87e9407f208e7fac285d49964703f47d97ad95dad5: Status 404 returned error can't find the container with id 8aef130603fa74ae92cffd87e9407f208e7fac285d49964703f47d97ad95dad5
Nov 01 08:32:27.813612 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:27.813586    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f2d60e_838e_42df_8e42_58f036371709.slice/crio-ba9ecc109b26d84b992a3d4296796403e5fed5a8ea0854301edadf35ea1188f5 WatchSource:0}: Error finding container ba9ecc109b26d84b992a3d4296796403e5fed5a8ea0854301edadf35ea1188f5: Status 404 returned error can't find the container with id ba9ecc109b26d84b992a3d4296796403e5fed5a8ea0854301edadf35ea1188f5
Nov 01 08:32:27.813875 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:27.813854    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8hhs6"]
Nov 01 08:32:28.183250 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:28.183217    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c6bf599bf-9cjkf" event={"ID":"3ca1667a-32ed-4777-8755-ec7e72add722","Type":"ContainerStarted","Data":"72e4eaedb1be7cdbff51da061a0f86a00b86b7d6a679cd71444858bd8f43cac2"}
Nov 01 08:32:28.183708 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:28.183269    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c6bf599bf-9cjkf" event={"ID":"3ca1667a-32ed-4777-8755-ec7e72add722","Type":"ContainerStarted","Data":"8aef130603fa74ae92cffd87e9407f208e7fac285d49964703f47d97ad95dad5"}
Nov 01 08:32:28.184444 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:28.184415    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hhs6" event={"ID":"c6f2d60e-838e-42df-8e42-58f036371709","Type":"ContainerStarted","Data":"ba9ecc109b26d84b992a3d4296796403e5fed5a8ea0854301edadf35ea1188f5"}
Nov 01 08:32:28.185935 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:28.185905    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4g77m" event={"ID":"08abdd05-c01c-4c69-ac14-330e24384e83","Type":"ContainerStarted","Data":"6e6b50f36b9b52752512d975818b9d0add24659dfb2c269584f872ccd5d6156d"}
Nov 01 08:32:28.186018 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:28.185951    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4g77m" event={"ID":"08abdd05-c01c-4c69-ac14-330e24384e83","Type":"ContainerStarted","Data":"3f8bc9f3a791d01b80b0d6597e8a1ed3d59df6318feb8863f5f91ba354f9a93b"}
Nov 01 08:32:28.186982 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:28.186961    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6" event={"ID":"f1f77890-a449-4958-9b83-ef93acbe2f58","Type":"ContainerStarted","Data":"aecd15dc0fc12380b56d709d2db9e61c2dc409a33e2eef9165cbcbd12132d7fc"}
Nov 01 08:32:28.340558 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:28.340527    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:28.343354 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:28.343330    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:29.182453 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:29.181656    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78dd945d47-r7lx4"]
Nov 01 08:32:29.201834 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:29.201802    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4g77m" event={"ID":"08abdd05-c01c-4c69-ac14-330e24384e83","Type":"ContainerStarted","Data":"5292b812f3b2509487f967237814d5eea9bc91101587f72a178217382727c718"}
Nov 01 08:32:29.202358 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:29.201872    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:29.203227 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:29.203201    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c6bf599bf-9cjkf"
Nov 01 08:32:30.209217 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:30.209163    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hhs6" event={"ID":"c6f2d60e-838e-42df-8e42-58f036371709","Type":"ContainerStarted","Data":"76e293606e000690f9b58b4382e94f839010c3202e38b2cfad691d54f56020c3"}
Nov 01 08:32:30.213143 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:30.213087    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bksj6" event={"ID":"f1f77890-a449-4958-9b83-ef93acbe2f58","Type":"ContainerStarted","Data":"b64c0e4242220eef3ee6d773b5e0230b6e2691e7a3ecaed27a49e7548fc96243"}
Nov 01 08:32:30.258771 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:30.258739    4931 patch_prober.go:29] interesting pod/image-registry-6fd6466786-gpntg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 08:32:30.258909 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:30.258808    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 08:32:31.219004 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:31.218960    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4g77m" event={"ID":"08abdd05-c01c-4c69-ac14-330e24384e83","Type":"ContainerStarted","Data":"46081751b85b7567fa75aed1e67104293ca2ca64df67c5cded87d99fcfec2923"}
Nov 01 08:32:31.220892 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:31.220858    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hhs6" event={"ID":"c6f2d60e-838e-42df-8e42-58f036371709","Type":"ContainerStarted","Data":"73e7ceb9d30cc746872225670a34cb87bd1a10c6f27ce05b819accb1b342b695"}
Nov 01 08:32:31.221114 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:31.221099    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:32.474141 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:32.474109    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bj2zg"
Nov 01 08:32:33.193337 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:33.193301    4931 patch_prober.go:29] interesting pod/image-registry-6fd6466786-gpntg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 08:32:33.193579 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:33.193355    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 08:32:39.188144 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:39.188105    4931 patch_prober.go:29] interesting pod/image-registry-78dd945d47-r7lx4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 08:32:39.188530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:39.188162    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4" podUID="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 08:32:39.266895 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:39.266856    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mnsws"
Nov 01 08:32:40.257906 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:40.257875    4931 patch_prober.go:29] interesting pod/image-registry-6fd6466786-gpntg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 08:32:40.258322 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:40.257924    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 08:32:41.226749 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:41.226720    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8hhs6"
Nov 01 08:32:43.193631 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:43.193594    4931 patch_prober.go:29] interesting pod/image-registry-6fd6466786-gpntg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 08:32:43.194015 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:43.193660    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 08:32:46.152324 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.152287    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-td2fk"]
Nov 01 08:32:46.190098 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.190072    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.292918 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.292881    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 08:32:46.292918 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.292881    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 08:32:46.324642 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.324611    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 08:32:46.324811 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.324676    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qwt9q\""
Nov 01 08:32:46.324811 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.324724    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 08:32:46.324910 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.324888    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.324946 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.324922    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6kjw\" (UniqueName: \"kubernetes.io/projected/5bdd19c7-7192-4fb7-bc9c-85438de618a3-kube-api-access-h6kjw\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.325003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.324951    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-wtmp\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.325064 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.325050    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.325102 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.325077    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-textfile\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.325152 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.325110    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-tls\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.325190 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.325163    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-sys\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.325241 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.325190    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bdd19c7-7192-4fb7-bc9c-85438de618a3-metrics-client-ca\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.325241 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.325233    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-root\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426216 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426141    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-sys\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426216 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426172    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bdd19c7-7192-4fb7-bc9c-85438de618a3-metrics-client-ca\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426216 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426207    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-root\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426245    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426249    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-sys\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426262    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6kjw\" (UniqueName: \"kubernetes.io/projected/5bdd19c7-7192-4fb7-bc9c-85438de618a3-kube-api-access-h6kjw\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426302    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-wtmp\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426322    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426347    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-textfile\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426353    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-root\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426412    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-tls\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426448    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-wtmp\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426922 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426737    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-textfile\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426922 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426807    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bdd19c7-7192-4fb7-bc9c-85438de618a3-metrics-client-ca\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.426922 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.426900    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.428781 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.428761    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-tls\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.428876 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.428807    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bdd19c7-7192-4fb7-bc9c-85438de618a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.488038 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.487994    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6kjw\" (UniqueName: \"kubernetes.io/projected/5bdd19c7-7192-4fb7-bc9c-85438de618a3-kube-api-access-h6kjw\") pod \"node-exporter-td2fk\" (UID: \"5bdd19c7-7192-4fb7-bc9c-85438de618a3\") " pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.538966 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:46.538920    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-td2fk"
Nov 01 08:32:46.578567 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:32:46.578534    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdd19c7_7192_4fb7_bc9c_85438de618a3.slice/crio-db9d89fbe30f3085c9a148b7ab5e65a5c1e1c388faa50db7fafdde925c101c84 WatchSource:0}: Error finding container db9d89fbe30f3085c9a148b7ab5e65a5c1e1c388faa50db7fafdde925c101c84: Status 404 returned error can't find the container with id db9d89fbe30f3085c9a148b7ab5e65a5c1e1c388faa50db7fafdde925c101c84
Nov 01 08:32:47.278801 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:47.278765    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-td2fk" event={"ID":"5bdd19c7-7192-4fb7-bc9c-85438de618a3","Type":"ContainerStarted","Data":"db9d89fbe30f3085c9a148b7ab5e65a5c1e1c388faa50db7fafdde925c101c84"}
Nov 01 08:32:48.284662 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:48.284574    4931 generic.go:356] "Generic (PLEG): container finished" podID="5bdd19c7-7192-4fb7-bc9c-85438de618a3" containerID="2e7422ecd714c20d46f7d5f450d2417ab5f2bc61005c7ceeb37acc1bf886132f" exitCode=0
Nov 01 08:32:48.285119 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:48.284660    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-td2fk" event={"ID":"5bdd19c7-7192-4fb7-bc9c-85438de618a3","Type":"ContainerDied","Data":"2e7422ecd714c20d46f7d5f450d2417ab5f2bc61005c7ceeb37acc1bf886132f"}
Nov 01 08:32:49.186778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:49.186746    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:49.290206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:49.290127    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-td2fk" event={"ID":"5bdd19c7-7192-4fb7-bc9c-85438de618a3","Type":"ContainerStarted","Data":"52b3364659f674a15c9992106945e080158c6974ee333b2d1a3798fbd5729875"}
Nov 01 08:32:49.290206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:49.290184    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-td2fk" event={"ID":"5bdd19c7-7192-4fb7-bc9c-85438de618a3","Type":"ContainerStarted","Data":"cfa2247e8da6e147dd5add3e50889a50e72ae983544c183e344f80e76510c2e1"}
Nov 01 08:32:49.465423 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:49.465317    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-td2fk" podStartSLOduration=1.990274962 podStartE2EDuration="3.465303561s" totalImagesPullingTime="1.400278828s" totalInitContainerRuntime="74.749771ms" isStatefulPod=true podCreationTimestamp="2026-11-01 08:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:49.46451891 +0000 UTC m=+39.579391879" watchObservedRunningTime="2026-11-01 08:32:49.465303561 +0000 UTC m=+39.580176528"
Nov 01 08:32:50.257480 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:50.257454    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:32:51.202515 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:51.202483    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fd6466786-gpntg"]
Nov 01 08:32:54.207761 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.207703    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4" podUID="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" containerName="registry" containerID="cri-o://794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b" gracePeriod=30
Nov 01 08:32:54.459816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.459748    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:54.495070 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495016    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets\") pod \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") "
Nov 01 08:32:54.495244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495110    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca\") pod \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") "
Nov 01 08:32:54.495244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495137    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration\") pod \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") "
Nov 01 08:32:54.495244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495163    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") pod \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") "
Nov 01 08:32:54.495244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495182    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted\") pod \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") "
Nov 01 08:32:54.495244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495204    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates\") pod \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") "
Nov 01 08:32:54.495244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495237    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p\") pod \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") "
Nov 01 08:32:54.495529 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495267    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token\") pod \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\" (UID: \"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab\") "
Nov 01 08:32:54.495819 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.495773    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca" pod "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:32:54.496255 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.496175    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates" pod "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:32:54.498079 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.498045    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration" pod "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:32:54.498182 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.498078    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p" pod "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab"). InnerVolumeSpecName "kube-api-access-fpw5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:32:54.498182 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.498113    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets" pod "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:32:54.498275 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.498246    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token" pod "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:32:54.498343 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.498318    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls" pod "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:32:54.502995 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.502972    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted" pod "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" (UID: "5974cc8f-4fe4-4143-bd79-d1b8888fc1ab"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:32:54.596081 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.596043    4931 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-trusted-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:32:54.596081 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.596077    4931 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-image-registry-private-configuration\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:32:54.596081 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.596091    4931 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:32:54.596340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.596103    4931 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-ca-trust-extracted\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:32:54.596340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.596113    4931 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-registry-certificates\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:32:54.596340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.596124    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpw5p\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-kube-api-access-fpw5p\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:32:54.596340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.596133    4931 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-bound-sa-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:32:54.596340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:54.596141    4931 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab-installation-pull-secrets\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:32:55.319423 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.319387    4931 generic.go:356] "Generic (PLEG): container finished" podID="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" containerID="794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b" exitCode=0
Nov 01 08:32:55.319886 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.319444    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4" event={"ID":"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab","Type":"ContainerDied","Data":"794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b"}
Nov 01 08:32:55.319886 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.319450    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4"
Nov 01 08:32:55.319886 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.319492    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dd945d47-r7lx4" event={"ID":"5974cc8f-4fe4-4143-bd79-d1b8888fc1ab","Type":"ContainerDied","Data":"09c8697b50e517d3b2ff8b31b19482bb0c405f77dce2b365ae5412d108181fd6"}
Nov 01 08:32:55.319886 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.319512    4931 scope.go:122] "RemoveContainer" containerID="794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b"
Nov 01 08:32:55.330958 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.330934    4931 scope.go:122] "RemoveContainer" containerID="794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b"
Nov 01 08:32:55.331315 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:32:55.331288    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b\": container with ID starting with 794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b not found: ID does not exist" containerID="794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b"
Nov 01 08:32:55.331376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.331329    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b"} err="failed to get container status \"794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b\": rpc error: code = NotFound desc = could not find container \"794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b\": container with ID starting with 794505764c917de468576fcd04da9adc12f7d25f10cca180d6b51c68f5caf62b not found: ID does not exist"
Nov 01 08:32:55.524441 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.524405    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78dd945d47-r7lx4"]
Nov 01 08:32:55.534766 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:55.534739    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78dd945d47-r7lx4"]
Nov 01 08:32:56.046180 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:32:56.046142    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" path="/var/lib/kubelet/pods/5974cc8f-4fe4-4143-bd79-d1b8888fc1ab/volumes"
Nov 01 08:33:01.024273 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:33:01.024232    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:33:16.222656 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.222607    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerName="registry" containerID="cri-o://922f008f848046a955b3d4d188a870c8227e882810b2e86a20c83b7a13d55e78" gracePeriod=30
Nov 01 08:33:16.401903 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.401868    4931 generic.go:356] "Generic (PLEG): container finished" podID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerID="922f008f848046a955b3d4d188a870c8227e882810b2e86a20c83b7a13d55e78" exitCode=0
Nov 01 08:33:16.402071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.401917    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" event={"ID":"6ddfd7c8-e096-40dc-8e3d-a359cb566176","Type":"ContainerDied","Data":"922f008f848046a955b3d4d188a870c8227e882810b2e86a20c83b7a13d55e78"}
Nov 01 08:33:16.481509 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.481450    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:33:16.573335 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573301    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb\") pod \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") "
Nov 01 08:33:16.573581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573351    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") pod \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") "
Nov 01 08:33:16.573581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573390    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration\") pod \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") "
Nov 01 08:33:16.573581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573413    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets\") pod \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") "
Nov 01 08:33:16.573581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573439    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates\") pod \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") "
Nov 01 08:33:16.573581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573523    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted\") pod \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") "
Nov 01 08:33:16.573581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573554    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca\") pod \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") "
Nov 01 08:33:16.573894 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573601    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token\") pod \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\" (UID: \"6ddfd7c8-e096-40dc-8e3d-a359cb566176\") "
Nov 01 08:33:16.574071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.573990    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates" pod "6ddfd7c8-e096-40dc-8e3d-a359cb566176" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:16.574580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.574529    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca" pod "6ddfd7c8-e096-40dc-8e3d-a359cb566176" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:33:16.576225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.576187    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token" pod "6ddfd7c8-e096-40dc-8e3d-a359cb566176" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:33:16.576381 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.576326    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets" pod "6ddfd7c8-e096-40dc-8e3d-a359cb566176" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:16.576444 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.576424    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration" pod "6ddfd7c8-e096-40dc-8e3d-a359cb566176" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:33:16.576539 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.576519    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls" pod "6ddfd7c8-e096-40dc-8e3d-a359cb566176" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:33:16.576588 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.576525    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb" pod "6ddfd7c8-e096-40dc-8e3d-a359cb566176" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176"). InnerVolumeSpecName "kube-api-access-t5xkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:33:16.581489 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.581381    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted" pod "6ddfd7c8-e096-40dc-8e3d-a359cb566176" (UID: "6ddfd7c8-e096-40dc-8e3d-a359cb566176"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:33:16.674695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.674664    4931 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-bound-sa-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:16.674695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.674692    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5xkb\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-kube-api-access-t5xkb\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:16.674695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.674703    4931 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:16.674938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.674714    4931 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-image-registry-private-configuration\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:16.674938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.674724    4931 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ddfd7c8-e096-40dc-8e3d-a359cb566176-installation-pull-secrets\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:16.674938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.674732    4931 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-registry-certificates\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:16.674938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.674741    4931 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ddfd7c8-e096-40dc-8e3d-a359cb566176-ca-trust-extracted\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:16.674938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:16.674750    4931 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfd7c8-e096-40dc-8e3d-a359cb566176-trusted-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:33:17.407113 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:17.407079    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd6466786-gpntg" event={"ID":"6ddfd7c8-e096-40dc-8e3d-a359cb566176","Type":"ContainerDied","Data":"f401ca95eb5d256f6b3064342e2d37fd627fbb4a796e082831d15c52e2ec0164"}
Nov 01 08:33:17.407546 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:17.407138    4931 scope.go:122] "RemoveContainer" containerID="922f008f848046a955b3d4d188a870c8227e882810b2e86a20c83b7a13d55e78"
Nov 01 08:33:17.407546 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:17.407088    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd6466786-gpntg"
Nov 01 08:33:17.612364 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:17.612327    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fd6466786-gpntg"]
Nov 01 08:33:17.660017 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:17.659937    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fd6466786-gpntg"]
Nov 01 08:33:18.045699 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:18.045625    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" path="/var/lib/kubelet/pods/6ddfd7c8-e096-40dc-8e3d-a359cb566176/volumes"
Nov 01 08:33:20.420872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:20.420792    4931 generic.go:356] "Generic (PLEG): container finished" podID="81ea361a-8d47-4aee-8ed8-b742628152ba" containerID="88ebb616a07f53bab13aca4dc7a14dceab4812da00e9382ae0689c3898125851" exitCode=0
Nov 01 08:33:20.420872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:20.420856    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-62p6x" event={"ID":"81ea361a-8d47-4aee-8ed8-b742628152ba","Type":"ContainerDied","Data":"88ebb616a07f53bab13aca4dc7a14dceab4812da00e9382ae0689c3898125851"}
Nov 01 08:33:20.421296 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:20.421192    4931 scope.go:122] "RemoveContainer" containerID="88ebb616a07f53bab13aca4dc7a14dceab4812da00e9382ae0689c3898125851"
Nov 01 08:33:21.425949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:21.425910    4931 generic.go:356] "Generic (PLEG): container finished" podID="2023ca6f-9936-4e4c-80a1-e33b47fa7dcb" containerID="6a135f829f5c44cf468f94bb5bb7c22782ae90a46d57dfc7dfcab0c6eb465a38" exitCode=0
Nov 01 08:33:21.426434 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:21.425982    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8" event={"ID":"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb","Type":"ContainerDied","Data":"6a135f829f5c44cf468f94bb5bb7c22782ae90a46d57dfc7dfcab0c6eb465a38"}
Nov 01 08:33:21.426434 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:21.426376    4931 scope.go:122] "RemoveContainer" containerID="6a135f829f5c44cf468f94bb5bb7c22782ae90a46d57dfc7dfcab0c6eb465a38"
Nov 01 08:33:21.427887 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:21.427863    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-62p6x" event={"ID":"81ea361a-8d47-4aee-8ed8-b742628152ba","Type":"ContainerStarted","Data":"0bc5defabcfc0fb94bc8d9c54caf21e1300474fa1f90a47ab43279b1c9dd097d"}
Nov 01 08:33:22.434371 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:22.434333    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9lnx8" event={"ID":"2023ca6f-9936-4e4c-80a1-e33b47fa7dcb","Type":"ContainerStarted","Data":"99e37465122e281596918e8c423b6b7bcc126a4aaf95ab1e84a6b94a9ccd43ee"}
Nov 01 08:33:31.021682 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:33:31.021646    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:33:35.484203 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:35.484173    4931 generic.go:356] "Generic (PLEG): container finished" podID="ae17519f-2301-43ce-ac7e-890042574c55" containerID="60eef37798274a5c29e3dcb1c94118de1ca965856ed1b1ae63130558e0d29ad6" exitCode=0
Nov 01 08:33:35.484554 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:35.484229    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4" event={"ID":"ae17519f-2301-43ce-ac7e-890042574c55","Type":"ContainerDied","Data":"60eef37798274a5c29e3dcb1c94118de1ca965856ed1b1ae63130558e0d29ad6"}
Nov 01 08:33:35.484554 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:35.484536    4931 scope.go:122] "RemoveContainer" containerID="60eef37798274a5c29e3dcb1c94118de1ca965856ed1b1ae63130558e0d29ad6"
Nov 01 08:33:36.490064 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:33:36.490009    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f8xd4" event={"ID":"ae17519f-2301-43ce-ac7e-890042574c55","Type":"ContainerStarted","Data":"e64d8fe8443384aa1cb255945ec23e31b6d9766b4c20931f2983702f15d75952"}
Nov 01 08:34:01.021016 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:34:01.020967    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:34:31.021911 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:34:31.021821    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:35:01.022882 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:35:01.022839    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:35:31.021080 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:35:31.021045    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:36:01.022150 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:36:01.022068    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:36:31.021735 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:36:31.021695    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:37:01.022805 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:37:01.022765    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:37:10.048975 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:10.048948    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:37:10.050660 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:10.050637    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:37:10.052519 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:10.052493    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:37:10.054169 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:10.054149    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:37:11.397223 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.397187    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"]
Nov 01 08:37:11.397619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.397519    4931 cpu_manager.go:414] "containerMap: removing container" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerName="registry"
Nov 01 08:37:11.397619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.397532    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerName="registry"
Nov 01 08:37:11.397619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.397544    4931 cpu_manager.go:414] "containerMap: removing container" podUID="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" containerName="registry"
Nov 01 08:37:11.397619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.397548    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" containerName="registry"
Nov 01 08:37:11.397619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.397607    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ddfd7c8-e096-40dc-8e3d-a359cb566176" containerName="registry"
Nov 01 08:37:11.397619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.397614    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5974cc8f-4fe4-4143-bd79-d1b8888fc1ab" containerName="registry"
Nov 01 08:37:11.400707 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.400686    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.471048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.470982    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4sv\" (UniqueName: \"kubernetes.io/projected/eb971946-a0da-4af8-92d9-778b7e0dc936-kube-api-access-nc4sv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.471048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.471044    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.471244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.471153    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.498460 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.498427    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 08:37:11.498596 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.498512    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-8mn7l\""
Nov 01 08:37:11.498634 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.498607    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 08:37:11.499730 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.499709    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"]
Nov 01 08:37:11.502959 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.502938    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"]
Nov 01 08:37:11.506316 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.506300    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.556945 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.556913    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pc49r\""
Nov 01 08:37:11.565895 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.565868    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"]
Nov 01 08:37:11.572083 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.572052    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.572225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.572126    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.572225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.572186    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.572310 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.572294    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4sv\" (UniqueName: \"kubernetes.io/projected/eb971946-a0da-4af8-92d9-778b7e0dc936-kube-api-access-nc4sv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.572362 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.572331    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.572412 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.572360    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmw2\" (UniqueName: \"kubernetes.io/projected/a51cc3f7-25a8-4ed3-844c-8effab164046-kube-api-access-hqmw2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.572412 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.572378    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.572603 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.572586    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.615445 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.615402    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"]
Nov 01 08:37:11.618973 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.618946    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.631135 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.631108    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4sv\" (UniqueName: \"kubernetes.io/projected/eb971946-a0da-4af8-92d9-778b7e0dc936-kube-api-access-nc4sv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.669462 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.669368    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-fq9vb\""
Nov 01 08:37:11.672954 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.672928    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.673090 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.672975    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmw2\" (UniqueName: \"kubernetes.io/projected/a51cc3f7-25a8-4ed3-844c-8effab164046-kube-api-access-hqmw2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.673090 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.673016    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.673090 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.673070    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.673268 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.673214    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.673320 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.673271    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck4kd\" (UniqueName: \"kubernetes.io/projected/edf8d7aa-a246-4017-af57-47b322142802-kube-api-access-ck4kd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.673444 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.673418    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.673735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.673651    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.674264 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.674246    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"]
Nov 01 08:37:11.700794 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.700754    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"]
Nov 01 08:37:11.704280 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.704256    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.716321 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.716300    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:11.732392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.732361    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmw2\" (UniqueName: \"kubernetes.io/projected/a51cc3f7-25a8-4ed3-844c-8effab164046-kube-api-access-hqmw2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.759812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.759776    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"]
Nov 01 08:37:11.771132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.770356    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-mlfjl\""
Nov 01 08:37:11.774437 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.774411    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.774531 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.774454    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.774531 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.774480    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.774531 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.774499    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.774685 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.774530    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck4kd\" (UniqueName: \"kubernetes.io/projected/edf8d7aa-a246-4017-af57-47b322142802-kube-api-access-ck4kd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.774685 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.774588    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dtn\" (UniqueName: \"kubernetes.io/projected/7049ff2a-c9c3-418f-b3ef-28b035ea297d-kube-api-access-k4dtn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.775071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.775051    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.775262 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.775247    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.821474 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.821442    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:11.834612 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.834576    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck4kd\" (UniqueName: \"kubernetes.io/projected/edf8d7aa-a246-4017-af57-47b322142802-kube-api-access-ck4kd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.875196 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.875166    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dtn\" (UniqueName: \"kubernetes.io/projected/7049ff2a-c9c3-418f-b3ef-28b035ea297d-kube-api-access-k4dtn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.875301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.875210    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.875301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.875238    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.875593 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.875577    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.876148 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.876122    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:11.934713 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.934637    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:11.935066 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:11.935047    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dtn\" (UniqueName: \"kubernetes.io/projected/7049ff2a-c9c3-418f-b3ef-28b035ea297d-kube-api-access-k4dtn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:12.020357 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.020327    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:12.061837 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.061805    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"]
Nov 01 08:37:12.065545 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.065524    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.087008 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.086891    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"]
Nov 01 08:37:12.089193 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:12.089157    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb971946_a0da_4af8_92d9_778b7e0dc936.slice/crio-165719c84094836e0c61885f6e65dbea3ed3a225e61704bd771235c6028119a6 WatchSource:0}: Error finding container 165719c84094836e0c61885f6e65dbea3ed3a225e61704bd771235c6028119a6: Status 404 returned error can't find the container with id 165719c84094836e0c61885f6e65dbea3ed3a225e61704bd771235c6028119a6
Nov 01 08:37:12.091151 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.091131    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:37:12.115979 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.115953    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-wnz7q\""
Nov 01 08:37:12.120818 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.120784    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"]
Nov 01 08:37:12.177845 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.177809    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.177845 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.177848    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8j7q\" (UniqueName: \"kubernetes.io/projected/24c9b3fa-c42f-4007-8516-c101d9a1300b-kube-api-access-d8j7q\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.178087 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.177874    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.187453 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.187377    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"]
Nov 01 08:37:12.190592 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:12.190563    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51cc3f7_25a8_4ed3_844c_8effab164046.slice/crio-4e43a6d965d57bf51dd379d46e314d0158602bd2d8e8cead6582ac07e06c8b03 WatchSource:0}: Error finding container 4e43a6d965d57bf51dd379d46e314d0158602bd2d8e8cead6582ac07e06c8b03: Status 404 returned error can't find the container with id 4e43a6d965d57bf51dd379d46e314d0158602bd2d8e8cead6582ac07e06c8b03
Nov 01 08:37:12.242836 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.242798    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9" event={"ID":"a51cc3f7-25a8-4ed3-844c-8effab164046","Type":"ContainerStarted","Data":"4e43a6d965d57bf51dd379d46e314d0158602bd2d8e8cead6582ac07e06c8b03"}
Nov 01 08:37:12.243941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.243917    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7" event={"ID":"eb971946-a0da-4af8-92d9-778b7e0dc936","Type":"ContainerStarted","Data":"165719c84094836e0c61885f6e65dbea3ed3a225e61704bd771235c6028119a6"}
Nov 01 08:37:12.279295 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.279264    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.279450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.279305    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8j7q\" (UniqueName: \"kubernetes.io/projected/24c9b3fa-c42f-4007-8516-c101d9a1300b-kube-api-access-d8j7q\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.279450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.279331    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.279718 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.279698    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.279754 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.279700    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.300088 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.300062    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"]
Nov 01 08:37:12.302240 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:12.302213    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf8d7aa_a246_4017_af57_47b322142802.slice/crio-9d26cf9be29848cb8043d7431c34491f6222a30d35d2a2e8d98638ed76347a8e WatchSource:0}: Error finding container 9d26cf9be29848cb8043d7431c34491f6222a30d35d2a2e8d98638ed76347a8e: Status 404 returned error can't find the container with id 9d26cf9be29848cb8043d7431c34491f6222a30d35d2a2e8d98638ed76347a8e
Nov 01 08:37:12.337955 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.337923    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8j7q\" (UniqueName: \"kubernetes.io/projected/24c9b3fa-c42f-4007-8516-c101d9a1300b-kube-api-access-d8j7q\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.387869 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.387846    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"]
Nov 01 08:37:12.388582 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.388564    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:12.389828 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:12.389802    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7049ff2a_c9c3_418f_b3ef_28b035ea297d.slice/crio-8f889a530f2bfa53ed5d60fde84cf539f2dc3ff3ee4bdbdbfcc685b3548c575b WatchSource:0}: Error finding container 8f889a530f2bfa53ed5d60fde84cf539f2dc3ff3ee4bdbdbfcc685b3548c575b: Status 404 returned error can't find the container with id 8f889a530f2bfa53ed5d60fde84cf539f2dc3ff3ee4bdbdbfcc685b3548c575b
Nov 01 08:37:12.753993 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:12.753972    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"]
Nov 01 08:37:12.756336 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:12.756307    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c9b3fa_c42f_4007_8516_c101d9a1300b.slice/crio-10449ac52ab4d44906dc4b7c69ca5b75a7563c9302827841aebcfdf85e483a1c WatchSource:0}: Error finding container 10449ac52ab4d44906dc4b7c69ca5b75a7563c9302827841aebcfdf85e483a1c: Status 404 returned error can't find the container with id 10449ac52ab4d44906dc4b7c69ca5b75a7563c9302827841aebcfdf85e483a1c
Nov 01 08:37:13.255290 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.255170    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq" event={"ID":"edf8d7aa-a246-4017-af57-47b322142802","Type":"ContainerStarted","Data":"9d26cf9be29848cb8043d7431c34491f6222a30d35d2a2e8d98638ed76347a8e"}
Nov 01 08:37:13.259669 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.259628    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr" event={"ID":"24c9b3fa-c42f-4007-8516-c101d9a1300b","Type":"ContainerStarted","Data":"10449ac52ab4d44906dc4b7c69ca5b75a7563c9302827841aebcfdf85e483a1c"}
Nov 01 08:37:13.260741 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.260717    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7" event={"ID":"7049ff2a-c9c3-418f-b3ef-28b035ea297d","Type":"ContainerStarted","Data":"8f889a530f2bfa53ed5d60fde84cf539f2dc3ff3ee4bdbdbfcc685b3548c575b"}
Nov 01 08:37:13.691119 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.688846    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"]
Nov 01 08:37:13.694160 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.693340    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.792887 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.792834    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qkkp\" (UniqueName: \"kubernetes.io/projected/03d5ff8f-8b19-4a14-b00a-877324bb892d-kube-api-access-6qkkp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.793392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.792981    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.793392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.793076    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.795263 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.795239    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"]
Nov 01 08:37:13.795390 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.795365    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-bmfjn\""
Nov 01 08:37:13.894289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.894256    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.894482 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.894308    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qkkp\" (UniqueName: \"kubernetes.io/projected/03d5ff8f-8b19-4a14-b00a-877324bb892d-kube-api-access-6qkkp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.894482 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.894345    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.894680 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.894663    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.894849 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.894834    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:13.973038 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:13.972950    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qkkp\" (UniqueName: \"kubernetes.io/projected/03d5ff8f-8b19-4a14-b00a-877324bb892d-kube-api-access-6qkkp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:14.015952 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:14.015921    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:14.427052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:14.426989    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"]
Nov 01 08:37:14.430776 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:14.430738    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d5ff8f_8b19_4a14_b00a_877324bb892d.slice/crio-cfcb4e4197153ccd90a22ea54aa773c7bba125256cf73c9e5ad5df01df5cd96c WatchSource:0}: Error finding container cfcb4e4197153ccd90a22ea54aa773c7bba125256cf73c9e5ad5df01df5cd96c: Status 404 returned error can't find the container with id cfcb4e4197153ccd90a22ea54aa773c7bba125256cf73c9e5ad5df01df5cd96c
Nov 01 08:37:15.280062 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:15.280001    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f" event={"ID":"03d5ff8f-8b19-4a14-b00a-877324bb892d","Type":"ContainerStarted","Data":"cfcb4e4197153ccd90a22ea54aa773c7bba125256cf73c9e5ad5df01df5cd96c"}
Nov 01 08:37:16.224010 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.223973    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"]
Nov 01 08:37:16.228198 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.228167    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.282379 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.282349    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"]
Nov 01 08:37:16.315604 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.315570    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.315789 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.315617    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.315789 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.315722    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5j5j\" (UniqueName: \"kubernetes.io/projected/fcc224bc-2e02-44d6-a060-482e5c5ea6db-kube-api-access-r5j5j\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.329195 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.328249    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-db9jj\""
Nov 01 08:37:16.416605 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.416568    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.416814 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.416615    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.416814 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.416724    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5j5j\" (UniqueName: \"kubernetes.io/projected/fcc224bc-2e02-44d6-a060-482e5c5ea6db-kube-api-access-r5j5j\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.417498 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.417466    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.417738 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.417722    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.487931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.487856    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5j5j\" (UniqueName: \"kubernetes.io/projected/fcc224bc-2e02-44d6-a060-482e5c5ea6db-kube-api-access-r5j5j\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.549168 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.549127    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:16.960004 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:16.959958    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"]
Nov 01 08:37:16.971061 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:16.966585    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc224bc_2e02_44d6_a060_482e5c5ea6db.slice/crio-bf149f9ca2d481f6ab32562d917c9468f85554b22544a871dbf5f07879c7e6d9 WatchSource:0}: Error finding container bf149f9ca2d481f6ab32562d917c9468f85554b22544a871dbf5f07879c7e6d9: Status 404 returned error can't find the container with id bf149f9ca2d481f6ab32562d917c9468f85554b22544a871dbf5f07879c7e6d9
Nov 01 08:37:17.300780 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:17.300698    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l" event={"ID":"fcc224bc-2e02-44d6-a060-482e5c5ea6db","Type":"ContainerStarted","Data":"bf149f9ca2d481f6ab32562d917c9468f85554b22544a871dbf5f07879c7e6d9"}
Nov 01 08:37:18.132596 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.132557    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"]
Nov 01 08:37:18.137933 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.137341    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.187970 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.187924    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 08:37:18.206373 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.206327    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kpfx2\""
Nov 01 08:37:18.235572 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.235509    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.236222 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.236102    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvqzd\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-kube-api-access-rvqzd\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.237428 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.236190    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.237587 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.237466    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.237587 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.237500    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.237587 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.237535    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.237587 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.237559    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aedd00-ab73-46a1-8ef7-11d6440a4706-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.237587 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.237586    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.237990 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.237671    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.242097 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.240158    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"]
Nov 01 08:37:18.339114 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339075    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.339577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339135    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.339577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339171    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvqzd\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-kube-api-access-rvqzd\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.339577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339201    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.339577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339259    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.339577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339284    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.339577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339308    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.339577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339329    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aedd00-ab73-46a1-8ef7-11d6440a4706-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.339577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.339352    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.340104 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.340079    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.341167 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.340523    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.341167 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.340766    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.341167 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.341048    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.341558 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.341522    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aedd00-ab73-46a1-8ef7-11d6440a4706-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.350226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.350178    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.368762 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.368627    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.402942 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.402842    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvqzd\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-kube-api-access-rvqzd\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.407765 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.407715    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-x8wfg\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.428690 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.428657    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"]
Nov 01 08:37:18.434233 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.434209    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.461769 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.461568    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:18.484754 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.484719    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-pr7fj\""
Nov 01 08:37:18.492979 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.492952    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"]
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541509    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541567    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541602    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541630    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4kzw\" (UniqueName: \"kubernetes.io/projected/0b60d387-603f-4171-ac01-7a17b3dfddad-kube-api-access-h4kzw\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541673    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541730    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541777    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b60d387-603f-4171-ac01-7a17b3dfddad-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541820    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.542073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.541852    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.642787 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.642742    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.642939 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.642908    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.643110 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.643010    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b60d387-603f-4171-ac01-7a17b3dfddad-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.643110 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.643086    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.643650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.643410    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.643650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.643479    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.643650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.643528    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.643650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.643560    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.643650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.643588    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4kzw\" (UniqueName: \"kubernetes.io/projected/0b60d387-603f-4171-ac01-7a17b3dfddad-kube-api-access-h4kzw\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.644532 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.644322    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.644949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.644918    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.645240 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.645219    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.645745 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.645722    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.646402 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.646379    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b60d387-603f-4171-ac01-7a17b3dfddad-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.653544 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.653483    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.656037 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.655925    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.704527 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.704465    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b60d387-603f-4171-ac01-7a17b3dfddad-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.705366 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.705275    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4kzw\" (UniqueName: \"kubernetes.io/projected/0b60d387-603f-4171-ac01-7a17b3dfddad-kube-api-access-h4kzw\") pod \"openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65\" (UID: \"0b60d387-603f-4171-ac01-7a17b3dfddad\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:18.750881 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:18.750527    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:19.009669 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:19.008620    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"]
Nov 01 08:37:19.013922 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:19.013885    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aedd00_ab73_46a1_8ef7_11d6440a4706.slice/crio-2ec94dfc48a98b87877a6066cccd8fec1852fe4a7412ee25eb38a728d1179097 WatchSource:0}: Error finding container 2ec94dfc48a98b87877a6066cccd8fec1852fe4a7412ee25eb38a728d1179097: Status 404 returned error can't find the container with id 2ec94dfc48a98b87877a6066cccd8fec1852fe4a7412ee25eb38a728d1179097
Nov 01 08:37:19.154702 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:19.154569    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"]
Nov 01 08:37:19.332877 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:19.332692    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" event={"ID":"91aedd00-ab73-46a1-8ef7-11d6440a4706","Type":"ContainerStarted","Data":"2ec94dfc48a98b87877a6066cccd8fec1852fe4a7412ee25eb38a728d1179097"}
Nov 01 08:37:19.338131 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:19.337996    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65" event={"ID":"0b60d387-603f-4171-ac01-7a17b3dfddad","Type":"ContainerStarted","Data":"0262a53a906e18a0a4b5cd6a84a00f1df0e08411396f4ce632550106ae6d17d5"}
Nov 01 08:37:28.419825 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.419781    4931 generic.go:356] "Generic (PLEG): container finished" podID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerID="e148f9022bf55aeee3c8d62470df46c09933f02c993af8973ddd9b407f2930fc" exitCode=0
Nov 01 08:37:28.420352 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.419872    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9" event={"ID":"a51cc3f7-25a8-4ed3-844c-8effab164046","Type":"ContainerDied","Data":"e148f9022bf55aeee3c8d62470df46c09933f02c993af8973ddd9b407f2930fc"}
Nov 01 08:37:28.422754 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.422723    4931 generic.go:356] "Generic (PLEG): container finished" podID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerID="f1fe8e95a1d43ced6feef6401532b51d5d81a4c788fbedf08e49c0e33c0908c0" exitCode=0
Nov 01 08:37:28.422871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.422809    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7" event={"ID":"eb971946-a0da-4af8-92d9-778b7e0dc936","Type":"ContainerDied","Data":"f1fe8e95a1d43ced6feef6401532b51d5d81a4c788fbedf08e49c0e33c0908c0"}
Nov 01 08:37:28.427263 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.427237    4931 generic.go:356] "Generic (PLEG): container finished" podID="edf8d7aa-a246-4017-af57-47b322142802" containerID="49776279f8e02d216bd8eaeb1d893f73e8df49468225b3fd91c92315de588b3e" exitCode=0
Nov 01 08:37:28.427347 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.427328    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq" event={"ID":"edf8d7aa-a246-4017-af57-47b322142802","Type":"ContainerDied","Data":"49776279f8e02d216bd8eaeb1d893f73e8df49468225b3fd91c92315de588b3e"}
Nov 01 08:37:28.653282 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.653239    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:37:28.653412 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.653326    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:37:28.653412 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.653362    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:37:28.872680 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.872644    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:37:28.872832 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.872819    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:37:28.872939 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:28.872927    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:37:29.432700 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.432615    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65" event={"ID":"0b60d387-603f-4171-ac01-7a17b3dfddad","Type":"ContainerStarted","Data":"7513e5635f858ed9bde3d9b6bf3867afccbed6b58e3145891c8dd955f9374bb0"}
Nov 01 08:37:29.434143 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.434104    4931 generic.go:356] "Generic (PLEG): container finished" podID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerID="14c5f4e389758be65b0141e479706acc96760e38a7919857fdccc8cfc25f8756" exitCode=0
Nov 01 08:37:29.434256 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.434194    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr" event={"ID":"24c9b3fa-c42f-4007-8516-c101d9a1300b","Type":"ContainerDied","Data":"14c5f4e389758be65b0141e479706acc96760e38a7919857fdccc8cfc25f8756"}
Nov 01 08:37:29.435689 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.435661    4931 generic.go:356] "Generic (PLEG): container finished" podID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerID="d3ac352d73632e957704c30df0774600ebe1a17b2ff8cc1df4715aa5051991f3" exitCode=0
Nov 01 08:37:29.435779 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.435745    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7" event={"ID":"7049ff2a-c9c3-418f-b3ef-28b035ea297d","Type":"ContainerDied","Data":"d3ac352d73632e957704c30df0774600ebe1a17b2ff8cc1df4715aa5051991f3"}
Nov 01 08:37:29.437454 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.437376    4931 generic.go:356] "Generic (PLEG): container finished" podID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerID="b6f7799a161e378f5015c34a4c70dc48a64e6913e30f1646bbec94f2c943cd85" exitCode=0
Nov 01 08:37:29.437532 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.437451    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l" event={"ID":"fcc224bc-2e02-44d6-a060-482e5c5ea6db","Type":"ContainerDied","Data":"b6f7799a161e378f5015c34a4c70dc48a64e6913e30f1646bbec94f2c943cd85"}
Nov 01 08:37:29.439151 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.439130    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" event={"ID":"91aedd00-ab73-46a1-8ef7-11d6440a4706","Type":"ContainerStarted","Data":"78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0"}
Nov 01 08:37:29.440620 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.440603    4931 generic.go:356] "Generic (PLEG): container finished" podID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerID="2a5784ffc16413b022f1ef4f357710491c1df349a1c7e1b9f78a1f754631b9ec" exitCode=0
Nov 01 08:37:29.440712 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.440630    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f" event={"ID":"03d5ff8f-8b19-4a14-b00a-877324bb892d","Type":"ContainerDied","Data":"2a5784ffc16413b022f1ef4f357710491c1df349a1c7e1b9f78a1f754631b9ec"}
Nov 01 08:37:29.462442 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.462418    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:29.466849 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.466831    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:29.549607 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.549551    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65" podStartSLOduration=2.06503326 podStartE2EDuration="11.54953797s" totalImagesPullingTime="9.48450471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:29.548348508 +0000 UTC m=+319.663221478" watchObservedRunningTime="2026-11-01 08:37:29.54953797 +0000 UTC m=+319.664410929"
Nov 01 08:37:29.751440 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.751361    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:29.756105 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:29.756082    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:30.125661 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:30.125615    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podStartSLOduration=2.271012588 podStartE2EDuration="12.125601163s" totalImagesPullingTime="9.854588575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:30.124745254 +0000 UTC m=+320.239618222" watchObservedRunningTime="2026-11-01 08:37:30.125601163 +0000 UTC m=+320.240474130"
Nov 01 08:37:30.446753 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:30.446723    4931 generic.go:356] "Generic (PLEG): container finished" podID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerID="5b38fa13db095c6cd1f1ec0d353cd26e5d0f8acf48755628bd4795e714fc7b90" exitCode=0
Nov 01 08:37:30.447134 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:30.446802    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9" event={"ID":"a51cc3f7-25a8-4ed3-844c-8effab164046","Type":"ContainerDied","Data":"5b38fa13db095c6cd1f1ec0d353cd26e5d0f8acf48755628bd4795e714fc7b90"}
Nov 01 08:37:30.447379 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:30.447247    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:30.447379 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:30.447291    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:30.448114 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:30.448091    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65"
Nov 01 08:37:30.448204 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:30.448145    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:37:31.021418 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:37:31.021325    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:37:31.453361 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:31.453323    4931 generic.go:356] "Generic (PLEG): container finished" podID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerID="5a5120d228a0838f8af8825c933dd12f1f03d4e0b73d9d59b95dc212876109ea" exitCode=0
Nov 01 08:37:31.453800 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:31.453415    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7" event={"ID":"eb971946-a0da-4af8-92d9-778b7e0dc936","Type":"ContainerDied","Data":"5a5120d228a0838f8af8825c933dd12f1f03d4e0b73d9d59b95dc212876109ea"}
Nov 01 08:37:31.455536 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:31.455512    4931 generic.go:356] "Generic (PLEG): container finished" podID="edf8d7aa-a246-4017-af57-47b322142802" containerID="3eabb26f508c0632ce31558a6239779923df51eab5ea663e4d15fb8747d1396a" exitCode=0
Nov 01 08:37:31.455636 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:31.455589    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq" event={"ID":"edf8d7aa-a246-4017-af57-47b322142802","Type":"ContainerDied","Data":"3eabb26f508c0632ce31558a6239779923df51eab5ea663e4d15fb8747d1396a"}
Nov 01 08:37:31.457876 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:31.457853    4931 generic.go:356] "Generic (PLEG): container finished" podID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerID="ae87ffd39aa27b2a159f3b935677979c22071d2ed264763f43fd8f578d4b578b" exitCode=0
Nov 01 08:37:31.457971 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:31.457942    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7" event={"ID":"7049ff2a-c9c3-418f-b3ef-28b035ea297d","Type":"ContainerDied","Data":"ae87ffd39aa27b2a159f3b935677979c22071d2ed264763f43fd8f578d4b578b"}
Nov 01 08:37:31.460327 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:31.460225    4931 generic.go:356] "Generic (PLEG): container finished" podID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerID="14329ac4a69dc1a5dbe7dde951394a4f3c8055c66d1c7abad762444c008ea576" exitCode=0
Nov 01 08:37:31.460327 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:31.460293    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l" event={"ID":"fcc224bc-2e02-44d6-a060-482e5c5ea6db","Type":"ContainerDied","Data":"14329ac4a69dc1a5dbe7dde951394a4f3c8055c66d1c7abad762444c008ea576"}
Nov 01 08:37:34.486971 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:34.486852    4931 generic.go:356] "Generic (PLEG): container finished" podID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerID="7bbbb64ca48bfbb1118360ac0800056cd9ec4d73fdc3e7fce39e1437ad6c216e" exitCode=0
Nov 01 08:37:34.486971 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:34.486935    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr" event={"ID":"24c9b3fa-c42f-4007-8516-c101d9a1300b","Type":"ContainerDied","Data":"7bbbb64ca48bfbb1118360ac0800056cd9ec4d73fdc3e7fce39e1437ad6c216e"}
Nov 01 08:37:37.528813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:37.528002    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f" event={"ID":"03d5ff8f-8b19-4a14-b00a-877324bb892d","Type":"ContainerStarted","Data":"61fdb2745da0ee909317b2ee5ba0f4d2f0ac3072b3e50662be92db912d8048bd"}
Nov 01 08:37:38.540096 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:38.539645    4931 generic.go:356] "Generic (PLEG): container finished" podID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerID="61fdb2745da0ee909317b2ee5ba0f4d2f0ac3072b3e50662be92db912d8048bd" exitCode=0
Nov 01 08:37:38.540096 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:38.539793    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f" event={"ID":"03d5ff8f-8b19-4a14-b00a-877324bb892d","Type":"ContainerDied","Data":"61fdb2745da0ee909317b2ee5ba0f4d2f0ac3072b3e50662be92db912d8048bd"}
Nov 01 08:37:44.614811 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:44.614620    4931 generic.go:356] "Generic (PLEG): container finished" podID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerID="4952b88bf8ff79aacd43bed424259a68de462d3170f5cb11d379097829bc2bd3" exitCode=0
Nov 01 08:37:44.614811 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:44.614743    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9" event={"ID":"a51cc3f7-25a8-4ed3-844c-8effab164046","Type":"ContainerDied","Data":"4952b88bf8ff79aacd43bed424259a68de462d3170f5cb11d379097829bc2bd3"}
Nov 01 08:37:45.629980 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:45.629938    4931 generic.go:356] "Generic (PLEG): container finished" podID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerID="849842d82125f7a31941686abbfa9af158f9901629719b1b5252847e4fd785a0" exitCode=0
Nov 01 08:37:45.630674 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:45.630638    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr" event={"ID":"24c9b3fa-c42f-4007-8516-c101d9a1300b","Type":"ContainerDied","Data":"849842d82125f7a31941686abbfa9af158f9901629719b1b5252847e4fd785a0"}
Nov 01 08:37:45.850590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:45.850556    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:46.018176 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.018083    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-util\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-util\") pod \"a51cc3f7-25a8-4ed3-844c-8effab164046\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") "
Nov 01 08:37:46.018361 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.018182    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51cc3f7-25a8-4ed3-844c-8effab164046-kube-api-access-hqmw2\" (UniqueName: \"kubernetes.io/projected/a51cc3f7-25a8-4ed3-844c-8effab164046-kube-api-access-hqmw2\") pod \"a51cc3f7-25a8-4ed3-844c-8effab164046\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") "
Nov 01 08:37:46.018361 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.018224    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-bundle\") pod \"a51cc3f7-25a8-4ed3-844c-8effab164046\" (UID: \"a51cc3f7-25a8-4ed3-844c-8effab164046\") "
Nov 01 08:37:46.020374 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.020330    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-bundle" pod "a51cc3f7-25a8-4ed3-844c-8effab164046" (UID: "a51cc3f7-25a8-4ed3-844c-8effab164046"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:46.026090 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.026053    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-util" pod "a51cc3f7-25a8-4ed3-844c-8effab164046" (UID: "a51cc3f7-25a8-4ed3-844c-8effab164046"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:46.029059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.029003    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51cc3f7-25a8-4ed3-844c-8effab164046-kube-api-access-hqmw2" pod "a51cc3f7-25a8-4ed3-844c-8effab164046" (UID: "a51cc3f7-25a8-4ed3-844c-8effab164046"). InnerVolumeSpecName "kube-api-access-hqmw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:46.119177 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.119140    4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-util\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:46.119177 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.119177    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqmw2\" (UniqueName: \"kubernetes.io/projected/a51cc3f7-25a8-4ed3-844c-8effab164046-kube-api-access-hqmw2\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:46.119177 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.119193    4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a51cc3f7-25a8-4ed3-844c-8effab164046-bundle\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:46.639199 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.639156    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7" event={"ID":"7049ff2a-c9c3-418f-b3ef-28b035ea297d","Type":"ContainerStarted","Data":"878f70e9b728d5a57ab4af74d27647be4696d2dc140212d121e3d7fc9625b249"}
Nov 01 08:37:46.642888 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.642853    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9" event={"ID":"a51cc3f7-25a8-4ed3-844c-8effab164046","Type":"ContainerDied","Data":"4e43a6d965d57bf51dd379d46e314d0158602bd2d8e8cead6582ac07e06c8b03"}
Nov 01 08:37:46.642888 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.642875    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kmqw9"
Nov 01 08:37:46.643093 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.642909    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e43a6d965d57bf51dd379d46e314d0158602bd2d8e8cead6582ac07e06c8b03"
Nov 01 08:37:46.645618 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.645590    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7" event={"ID":"eb971946-a0da-4af8-92d9-778b7e0dc936","Type":"ContainerStarted","Data":"c9db79901b3d36b2306a5d365732c5d6647be427e4639f5b510b7f245e88c552"}
Nov 01 08:37:46.648183 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.648161    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq" event={"ID":"edf8d7aa-a246-4017-af57-47b322142802","Type":"ContainerStarted","Data":"9a3aee5283e8fb3998442f58af61f2906c1daca44b28ee1de1d8322a7004166a"}
Nov 01 08:37:46.796666 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.796611    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7" podStartSLOduration=2.848657624 podStartE2EDuration="35.796597107s" totalImagesPullingTime="32.947939483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:46.795231397 +0000 UTC m=+336.910104365" watchObservedRunningTime="2026-11-01 08:37:46.796597107 +0000 UTC m=+336.911470075"
Nov 01 08:37:46.823187 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.823157    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:46.908058 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.907987    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7" podStartSLOduration=2.771526556 podStartE2EDuration="35.907974864s" totalImagesPullingTime="33.136448308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:46.907568634 +0000 UTC m=+337.022441603" watchObservedRunningTime="2026-11-01 08:37:46.907974864 +0000 UTC m=+337.022847832"
Nov 01 08:37:46.928132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.928018    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-util\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-util\") pod \"24c9b3fa-c42f-4007-8516-c101d9a1300b\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") "
Nov 01 08:37:46.928132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.928121    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-bundle\") pod \"24c9b3fa-c42f-4007-8516-c101d9a1300b\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") "
Nov 01 08:37:46.928357 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.928198    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c9b3fa-c42f-4007-8516-c101d9a1300b-kube-api-access-d8j7q\" (UniqueName: \"kubernetes.io/projected/24c9b3fa-c42f-4007-8516-c101d9a1300b-kube-api-access-d8j7q\") pod \"24c9b3fa-c42f-4007-8516-c101d9a1300b\" (UID: \"24c9b3fa-c42f-4007-8516-c101d9a1300b\") "
Nov 01 08:37:46.928538 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.928509    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-bundle" pod "24c9b3fa-c42f-4007-8516-c101d9a1300b" (UID: "24c9b3fa-c42f-4007-8516-c101d9a1300b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:46.930570 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.930535    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c9b3fa-c42f-4007-8516-c101d9a1300b-kube-api-access-d8j7q" pod "24c9b3fa-c42f-4007-8516-c101d9a1300b" (UID: "24c9b3fa-c42f-4007-8516-c101d9a1300b"). InnerVolumeSpecName "kube-api-access-d8j7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:46.934400 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:46.934373    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-util" pod "24c9b3fa-c42f-4007-8516-c101d9a1300b" (UID: "24c9b3fa-c42f-4007-8516-c101d9a1300b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:47.029338 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.029303    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8j7q\" (UniqueName: \"kubernetes.io/projected/24c9b3fa-c42f-4007-8516-c101d9a1300b-kube-api-access-d8j7q\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:47.029338 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.029334    4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-util\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:47.029338 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.029343    4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24c9b3fa-c42f-4007-8516-c101d9a1300b-bundle\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:47.071656 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.071600    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq" podStartSLOduration=2.927648194 podStartE2EDuration="36.07158616s" totalImagesPullingTime="33.143937966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:47.071175693 +0000 UTC m=+337.186048664" watchObservedRunningTime="2026-11-01 08:37:47.07158616 +0000 UTC m=+337.186459127"
Nov 01 08:37:47.655215 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.655180    4931 generic.go:356] "Generic (PLEG): container finished" podID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerID="c9db79901b3d36b2306a5d365732c5d6647be427e4639f5b510b7f245e88c552" exitCode=0
Nov 01 08:37:47.655693 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.655249    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7" event={"ID":"eb971946-a0da-4af8-92d9-778b7e0dc936","Type":"ContainerDied","Data":"c9db79901b3d36b2306a5d365732c5d6647be427e4639f5b510b7f245e88c552"}
Nov 01 08:37:47.657202 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.657178    4931 generic.go:356] "Generic (PLEG): container finished" podID="edf8d7aa-a246-4017-af57-47b322142802" containerID="9a3aee5283e8fb3998442f58af61f2906c1daca44b28ee1de1d8322a7004166a" exitCode=0
Nov 01 08:37:47.657310 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.657257    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq" event={"ID":"edf8d7aa-a246-4017-af57-47b322142802","Type":"ContainerDied","Data":"9a3aee5283e8fb3998442f58af61f2906c1daca44b28ee1de1d8322a7004166a"}
Nov 01 08:37:47.659253 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.659231    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f" event={"ID":"03d5ff8f-8b19-4a14-b00a-877324bb892d","Type":"ContainerStarted","Data":"c15edd77284951992b208cfd24c85d5553f2b760178b7f6d44f0490c6f0d8f5c"}
Nov 01 08:37:47.660886 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.660868    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr"
Nov 01 08:37:47.660987 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.660867    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644427fr" event={"ID":"24c9b3fa-c42f-4007-8516-c101d9a1300b","Type":"ContainerDied","Data":"10449ac52ab4d44906dc4b7c69ca5b75a7563c9302827841aebcfdf85e483a1c"}
Nov 01 08:37:47.660987 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.660977    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10449ac52ab4d44906dc4b7c69ca5b75a7563c9302827841aebcfdf85e483a1c"
Nov 01 08:37:47.662891 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.662867    4931 generic.go:356] "Generic (PLEG): container finished" podID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerID="878f70e9b728d5a57ab4af74d27647be4696d2dc140212d121e3d7fc9625b249" exitCode=0
Nov 01 08:37:47.662994 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.662932    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7" event={"ID":"7049ff2a-c9c3-418f-b3ef-28b035ea297d","Type":"ContainerDied","Data":"878f70e9b728d5a57ab4af74d27647be4696d2dc140212d121e3d7fc9625b249"}
Nov 01 08:37:47.664973 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.664952    4931 generic.go:356] "Generic (PLEG): container finished" podID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerID="1d6977bee1f5cced1a4d8f991d77a681e239d095803974b0fca00063f9bbc706" exitCode=0
Nov 01 08:37:47.665090 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:47.664998    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l" event={"ID":"fcc224bc-2e02-44d6-a060-482e5c5ea6db","Type":"ContainerDied","Data":"1d6977bee1f5cced1a4d8f991d77a681e239d095803974b0fca00063f9bbc706"}
Nov 01 08:37:48.157465 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.157411    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f" podStartSLOduration=3.9826673169999998 podStartE2EDuration="35.157396708s" totalImagesPullingTime="31.174729391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:48.155891628 +0000 UTC m=+338.270764597" watchObservedRunningTime="2026-11-01 08:37:48.157396708 +0000 UTC m=+338.272269670"
Nov 01 08:37:48.671166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.671129    4931 generic.go:356] "Generic (PLEG): container finished" podID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerID="c15edd77284951992b208cfd24c85d5553f2b760178b7f6d44f0490c6f0d8f5c" exitCode=0
Nov 01 08:37:48.671565 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.671214    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f" event={"ID":"03d5ff8f-8b19-4a14-b00a-877324bb892d","Type":"ContainerDied","Data":"c15edd77284951992b208cfd24c85d5553f2b760178b7f6d44f0490c6f0d8f5c"}
Nov 01 08:37:48.815951 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.815922    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:48.877561 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.877538    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:48.880159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.880139    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:48.883656 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.883638    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:48.944533 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.944473    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-bundle\") pod \"edf8d7aa-a246-4017-af57-47b322142802\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") "
Nov 01 08:37:48.944533 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.944524    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-util\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-util\") pod \"edf8d7aa-a246-4017-af57-47b322142802\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") "
Nov 01 08:37:48.944684 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.944578    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf8d7aa-a246-4017-af57-47b322142802-kube-api-access-ck4kd\" (UniqueName: \"kubernetes.io/projected/edf8d7aa-a246-4017-af57-47b322142802-kube-api-access-ck4kd\") pod \"edf8d7aa-a246-4017-af57-47b322142802\" (UID: \"edf8d7aa-a246-4017-af57-47b322142802\") "
Nov 01 08:37:48.945190 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.945166    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-bundle" pod "edf8d7aa-a246-4017-af57-47b322142802" (UID: "edf8d7aa-a246-4017-af57-47b322142802"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:48.946763 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.946740    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf8d7aa-a246-4017-af57-47b322142802-kube-api-access-ck4kd" pod "edf8d7aa-a246-4017-af57-47b322142802" (UID: "edf8d7aa-a246-4017-af57-47b322142802"). InnerVolumeSpecName "kube-api-access-ck4kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:48.952677 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:48.952654    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-util" pod "edf8d7aa-a246-4017-af57-47b322142802" (UID: "edf8d7aa-a246-4017-af57-47b322142802"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:49.045203 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045176    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-util\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-util\") pod \"eb971946-a0da-4af8-92d9-778b7e0dc936\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") "
Nov 01 08:37:49.045383 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045209    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-bundle\") pod \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") "
Nov 01 08:37:49.045383 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045233    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-bundle\") pod \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") "
Nov 01 08:37:49.045383 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045272    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb971946-a0da-4af8-92d9-778b7e0dc936-kube-api-access-nc4sv\" (UniqueName: \"kubernetes.io/projected/eb971946-a0da-4af8-92d9-778b7e0dc936-kube-api-access-nc4sv\") pod \"eb971946-a0da-4af8-92d9-778b7e0dc936\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") "
Nov 01 08:37:49.045383 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045302    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-bundle\") pod \"eb971946-a0da-4af8-92d9-778b7e0dc936\" (UID: \"eb971946-a0da-4af8-92d9-778b7e0dc936\") "
Nov 01 08:37:49.045383 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045323    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7049ff2a-c9c3-418f-b3ef-28b035ea297d-kube-api-access-k4dtn\" (UniqueName: \"kubernetes.io/projected/7049ff2a-c9c3-418f-b3ef-28b035ea297d-kube-api-access-k4dtn\") pod \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") "
Nov 01 08:37:49.045383 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045349    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc224bc-2e02-44d6-a060-482e5c5ea6db-kube-api-access-r5j5j\" (UniqueName: \"kubernetes.io/projected/fcc224bc-2e02-44d6-a060-482e5c5ea6db-kube-api-access-r5j5j\") pod \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") "
Nov 01 08:37:49.045698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045387    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-util\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-util\") pod \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\" (UID: \"7049ff2a-c9c3-418f-b3ef-28b035ea297d\") "
Nov 01 08:37:49.045698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045490    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-util\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-util\") pod \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\" (UID: \"fcc224bc-2e02-44d6-a060-482e5c5ea6db\") "
Nov 01 08:37:49.045798 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045718    4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-bundle\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.045798 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045736    4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf8d7aa-a246-4017-af57-47b322142802-util\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.045798 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045748    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck4kd\" (UniqueName: \"kubernetes.io/projected/edf8d7aa-a246-4017-af57-47b322142802-kube-api-access-ck4kd\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.045946 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.045884    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-bundle" pod "fcc224bc-2e02-44d6-a060-482e5c5ea6db" (UID: "fcc224bc-2e02-44d6-a060-482e5c5ea6db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:49.046947 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.046310    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-bundle" pod "eb971946-a0da-4af8-92d9-778b7e0dc936" (UID: "eb971946-a0da-4af8-92d9-778b7e0dc936"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:49.048058 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.047980    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7049ff2a-c9c3-418f-b3ef-28b035ea297d-kube-api-access-k4dtn" pod "7049ff2a-c9c3-418f-b3ef-28b035ea297d" (UID: "7049ff2a-c9c3-418f-b3ef-28b035ea297d"). InnerVolumeSpecName "kube-api-access-k4dtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:49.048139 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.048105    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc224bc-2e02-44d6-a060-482e5c5ea6db-kube-api-access-r5j5j" pod "fcc224bc-2e02-44d6-a060-482e5c5ea6db" (UID: "fcc224bc-2e02-44d6-a060-482e5c5ea6db"). InnerVolumeSpecName "kube-api-access-r5j5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:49.048497 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.048476    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb971946-a0da-4af8-92d9-778b7e0dc936-kube-api-access-nc4sv" pod "eb971946-a0da-4af8-92d9-778b7e0dc936" (UID: "eb971946-a0da-4af8-92d9-778b7e0dc936"). InnerVolumeSpecName "kube-api-access-nc4sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:49.051469 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.051447    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-bundle" pod "7049ff2a-c9c3-418f-b3ef-28b035ea297d" (UID: "7049ff2a-c9c3-418f-b3ef-28b035ea297d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:49.052751 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.052733    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-util" pod "eb971946-a0da-4af8-92d9-778b7e0dc936" (UID: "eb971946-a0da-4af8-92d9-778b7e0dc936"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:49.053042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.053009    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-util" pod "fcc224bc-2e02-44d6-a060-482e5c5ea6db" (UID: "fcc224bc-2e02-44d6-a060-482e5c5ea6db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:49.055582 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.055558    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-util" pod "7049ff2a-c9c3-418f-b3ef-28b035ea297d" (UID: "7049ff2a-c9c3-418f-b3ef-28b035ea297d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:49.146376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146349    4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-util\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.146376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146371    4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-util\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.146376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146380    4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcc224bc-2e02-44d6-a060-482e5c5ea6db-bundle\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.146559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146390    4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-bundle\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.146559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146398    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc4sv\" (UniqueName: \"kubernetes.io/projected/eb971946-a0da-4af8-92d9-778b7e0dc936-kube-api-access-nc4sv\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.146559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146407    4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb971946-a0da-4af8-92d9-778b7e0dc936-bundle\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.146559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146416    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dtn\" (UniqueName: \"kubernetes.io/projected/7049ff2a-c9c3-418f-b3ef-28b035ea297d-kube-api-access-k4dtn\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.146559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146425    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5j5j\" (UniqueName: \"kubernetes.io/projected/fcc224bc-2e02-44d6-a060-482e5c5ea6db-kube-api-access-r5j5j\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.146559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.146433    4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7049ff2a-c9c3-418f-b3ef-28b035ea297d-util\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:49.677564 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.677524    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7" event={"ID":"7049ff2a-c9c3-418f-b3ef-28b035ea297d","Type":"ContainerDied","Data":"8f889a530f2bfa53ed5d60fde84cf539f2dc3ff3ee4bdbdbfcc685b3548c575b"}
Nov 01 08:37:49.678010 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.677574    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f889a530f2bfa53ed5d60fde84cf539f2dc3ff3ee4bdbdbfcc685b3548c575b"
Nov 01 08:37:49.678010 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.677532    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w9sc7"
Nov 01 08:37:49.679551 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.679527    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l"
Nov 01 08:37:49.679682 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.679553    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xt65l" event={"ID":"fcc224bc-2e02-44d6-a060-482e5c5ea6db","Type":"ContainerDied","Data":"bf149f9ca2d481f6ab32562d917c9468f85554b22544a871dbf5f07879c7e6d9"}
Nov 01 08:37:49.679682 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.679597    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf149f9ca2d481f6ab32562d917c9468f85554b22544a871dbf5f07879c7e6d9"
Nov 01 08:37:49.681558 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.681525    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7" event={"ID":"eb971946-a0da-4af8-92d9-778b7e0dc936","Type":"ContainerDied","Data":"165719c84094836e0c61885f6e65dbea3ed3a225e61704bd771235c6028119a6"}
Nov 01 08:37:49.681558 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.681546    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpmbh7"
Nov 01 08:37:49.681728 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.681577    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="165719c84094836e0c61885f6e65dbea3ed3a225e61704bd771235c6028119a6"
Nov 01 08:37:49.683447 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.683431    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq"
Nov 01 08:37:49.683561 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.683461    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at79jq" event={"ID":"edf8d7aa-a246-4017-af57-47b322142802","Type":"ContainerDied","Data":"9d26cf9be29848cb8043d7431c34491f6222a30d35d2a2e8d98638ed76347a8e"}
Nov 01 08:37:49.683561 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.683499    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d26cf9be29848cb8043d7431c34491f6222a30d35d2a2e8d98638ed76347a8e"
Nov 01 08:37:49.808406 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.808385    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:49.952529 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.952449    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-util\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-util\") pod \"03d5ff8f-8b19-4a14-b00a-877324bb892d\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") "
Nov 01 08:37:49.952529 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.952485    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d5ff8f-8b19-4a14-b00a-877324bb892d-kube-api-access-6qkkp\" (UniqueName: \"kubernetes.io/projected/03d5ff8f-8b19-4a14-b00a-877324bb892d-kube-api-access-6qkkp\") pod \"03d5ff8f-8b19-4a14-b00a-877324bb892d\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") "
Nov 01 08:37:49.952725 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.952590    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-bundle\") pod \"03d5ff8f-8b19-4a14-b00a-877324bb892d\" (UID: \"03d5ff8f-8b19-4a14-b00a-877324bb892d\") "
Nov 01 08:37:49.953331 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.953256    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-bundle" pod "03d5ff8f-8b19-4a14-b00a-877324bb892d" (UID: "03d5ff8f-8b19-4a14-b00a-877324bb892d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:49.954819 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.954788    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d5ff8f-8b19-4a14-b00a-877324bb892d-kube-api-access-6qkkp" pod "03d5ff8f-8b19-4a14-b00a-877324bb892d" (UID: "03d5ff8f-8b19-4a14-b00a-877324bb892d"). InnerVolumeSpecName "kube-api-access-6qkkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:37:49.957942 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:49.957921    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-util" pod "03d5ff8f-8b19-4a14-b00a-877324bb892d" (UID: "03d5ff8f-8b19-4a14-b00a-877324bb892d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:37:50.053242 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:50.053215    4931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-bundle\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:50.053242 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:50.053239    4931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d5ff8f-8b19-4a14-b00a-877324bb892d-util\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:50.053394 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:50.053249    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qkkp\" (UniqueName: \"kubernetes.io/projected/03d5ff8f-8b19-4a14-b00a-877324bb892d-kube-api-access-6qkkp\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:37:50.689789 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:50.689753    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f" event={"ID":"03d5ff8f-8b19-4a14-b00a-877324bb892d","Type":"ContainerDied","Data":"cfcb4e4197153ccd90a22ea54aa773c7bba125256cf73c9e5ad5df01df5cd96c"}
Nov 01 08:37:50.689789 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:50.689790    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbv2f"
Nov 01 08:37:50.690260 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:50.689802    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfcb4e4197153ccd90a22ea54aa773c7bba125256cf73c9e5ad5df01df5cd96c"
Nov 01 08:37:54.703523 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703487    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"]
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703814    4931 cpu_manager.go:414] "containerMap: removing container" podUID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerName="pull"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703825    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerName="pull"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703835    4931 cpu_manager.go:414] "containerMap: removing container" podUID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerName="extract"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703844    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerName="extract"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703855    4931 cpu_manager.go:414] "containerMap: removing container" podUID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerName="extract"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703864    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerName="extract"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703879    4931 cpu_manager.go:414] "containerMap: removing container" podUID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerName="util"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703884    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerName="util"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703891    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerName="extract"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703895    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerName="extract"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703903    4931 cpu_manager.go:414] "containerMap: removing container" podUID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerName="util"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703910    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerName="util"
Nov 01 08:37:54.703911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703920    4931 cpu_manager.go:414] "containerMap: removing container" podUID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703924    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703930    4931 cpu_manager.go:414] "containerMap: removing container" podUID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703934    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703939    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703945    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703953    4931 cpu_manager.go:414] "containerMap: removing container" podUID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703959    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703970    4931 cpu_manager.go:414] "containerMap: removing container" podUID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703975    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703983    4931 cpu_manager.go:414] "containerMap: removing container" podUID="edf8d7aa-a246-4017-af57-47b322142802" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.703990    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf8d7aa-a246-4017-af57-47b322142802" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704002    4931 cpu_manager.go:414] "containerMap: removing container" podUID="edf8d7aa-a246-4017-af57-47b322142802" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704009    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf8d7aa-a246-4017-af57-47b322142802" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704036    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704041    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704047    4931 cpu_manager.go:414] "containerMap: removing container" podUID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704051    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704059    4931 cpu_manager.go:414] "containerMap: removing container" podUID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704063    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704071    4931 cpu_manager.go:414] "containerMap: removing container" podUID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704074    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704084    4931 cpu_manager.go:414] "containerMap: removing container" podUID="edf8d7aa-a246-4017-af57-47b322142802" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704088    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf8d7aa-a246-4017-af57-47b322142802" containerName="util"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704094    4931 cpu_manager.go:414] "containerMap: removing container" podUID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704098    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704105    4931 cpu_manager.go:414] "containerMap: removing container" podUID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704110    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerName="pull"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704116    4931 cpu_manager.go:414] "containerMap: removing container" podUID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704119    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704169    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51cc3f7-25a8-4ed3-844c-8effab164046" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704178    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d5ff8f-8b19-4a14-b00a-877324bb892d" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704183    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf8d7aa-a246-4017-af57-47b322142802" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704190    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c9b3fa-c42f-4007-8516-c101d9a1300b" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704197    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc224bc-2e02-44d6-a060-482e5c5ea6db" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704204    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb971946-a0da-4af8-92d9-778b7e0dc936" containerName="extract"
Nov 01 08:37:54.704594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.704208    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7049ff2a-c9c3-418f-b3ef-28b035ea297d" containerName="extract"
Nov 01 08:37:54.723168 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.723142    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.773729 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.773701    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 08:37:54.773729 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.773709    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 08:37:54.774115 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.773825    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 08:37:54.774115 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.773894    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 08:37:54.790475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.790448    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.790747 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.790728    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e3b9fd-d920-439c-a748-2550f5a599b7-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.790931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.790916    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.791087 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.791072    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwgn\" (UniqueName: \"kubernetes.io/projected/48e3b9fd-d920-439c-a748-2550f5a599b7-kube-api-access-ncwgn\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.791244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.791223    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.791392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.791372    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e3b9fd-d920-439c-a748-2550f5a599b7-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.821506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.821477    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 08:37:54.821506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.821485    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-gx5kp\""
Nov 01 08:37:54.823754 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.823734    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"]
Nov 01 08:37:54.891999 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.891965    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.892185 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.892008    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwgn\" (UniqueName: \"kubernetes.io/projected/48e3b9fd-d920-439c-a748-2550f5a599b7-kube-api-access-ncwgn\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.892185 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.892066    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.892185 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.892104    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e3b9fd-d920-439c-a748-2550f5a599b7-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.892185 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.892145    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.892394 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.892191    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e3b9fd-d920-439c-a748-2550f5a599b7-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.892606 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.892572    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e3b9fd-d920-439c-a748-2550f5a599b7-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.894738 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.894706    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e3b9fd-d920-439c-a748-2550f5a599b7-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.894832 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.894768    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.894926 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.894909    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.894961 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.894928    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e3b9fd-d920-439c-a748-2550f5a599b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:54.951920 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:54.951892    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwgn\" (UniqueName: \"kubernetes.io/projected/48e3b9fd-d920-439c-a748-2550f5a599b7-kube-api-access-ncwgn\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8\" (UID: \"48e3b9fd-d920-439c-a748-2550f5a599b7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:55.072134 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:55.072052    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:37:55.568922 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:37:55.568895    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e3b9fd_d920_439c_a748_2550f5a599b7.slice/crio-a01442f059d2123c102a45ace25fc8246c61c01fa0989348faa0ea168931b9a0 WatchSource:0}: Error finding container a01442f059d2123c102a45ace25fc8246c61c01fa0989348faa0ea168931b9a0: Status 404 returned error can't find the container with id a01442f059d2123c102a45ace25fc8246c61c01fa0989348faa0ea168931b9a0
Nov 01 08:37:55.615264 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:55.615232    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"]
Nov 01 08:37:55.712885 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:37:55.712843    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8" event={"ID":"48e3b9fd-d920-439c-a748-2550f5a599b7","Type":"ContainerStarted","Data":"a01442f059d2123c102a45ace25fc8246c61c01fa0989348faa0ea168931b9a0"}
Nov 01 08:38:01.021868 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:38:01.021827    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:38:04.759643 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:04.759605    4931 generic.go:356] "Generic (PLEG): container finished" podID="48e3b9fd-d920-439c-a748-2550f5a599b7" containerID="b139a8b4c852442fefa12485736b1496cb4b639e23638a05e5ac365c922c6e4e" exitCode=0
Nov 01 08:38:04.760115 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:04.759683    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8" event={"ID":"48e3b9fd-d920-439c-a748-2550f5a599b7","Type":"ContainerDied","Data":"b139a8b4c852442fefa12485736b1496cb4b639e23638a05e5ac365c922c6e4e"}
Nov 01 08:38:05.765956 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:05.765920    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8" event={"ID":"48e3b9fd-d920-439c-a748-2550f5a599b7","Type":"ContainerStarted","Data":"11592bf85a4e9f9080db746f92f6a3e00791d870689f425b1ddbe1048db0c30c"}
Nov 01 08:38:05.767373 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:05.767336    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:38:05.961303 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:05.961273    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"]
Nov 01 08:38:05.963540 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:05.963522    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:05.995134 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:05.995103    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a2fd57-2c30-49d1-91f4-f461dde78747-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k86vt\" (UID: \"07a2fd57-2c30-49d1-91f4-f461dde78747\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:05.995264 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:05.995154    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hltv\" (UniqueName: \"kubernetes.io/projected/07a2fd57-2c30-49d1-91f4-f461dde78747-kube-api-access-4hltv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k86vt\" (UID: \"07a2fd57-2c30-49d1-91f4-f461dde78747\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:06.011204 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.011158    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8" podStartSLOduration=2.244408696 podStartE2EDuration="12.011145889s" totalImagesPullingTime="9.28546507s" totalInitContainerRuntime="481.272123ms" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:05.96380234 +0000 UTC m=+356.078675307" watchObservedRunningTime="2026-11-01 08:38:06.011145889 +0000 UTC m=+356.126018857"
Nov 01 08:38:06.018329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.018305    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 08:38:06.018329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.018318    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 08:38:06.018491 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.018389    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2vkbl\""
Nov 01 08:38:06.056509 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.056482    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"]
Nov 01 08:38:06.096577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.096542    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a2fd57-2c30-49d1-91f4-f461dde78747-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k86vt\" (UID: \"07a2fd57-2c30-49d1-91f4-f461dde78747\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:06.096738 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.096600    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hltv\" (UniqueName: \"kubernetes.io/projected/07a2fd57-2c30-49d1-91f4-f461dde78747-kube-api-access-4hltv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k86vt\" (UID: \"07a2fd57-2c30-49d1-91f4-f461dde78747\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:06.096902 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.096879    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a2fd57-2c30-49d1-91f4-f461dde78747-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k86vt\" (UID: \"07a2fd57-2c30-49d1-91f4-f461dde78747\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:06.155685 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.155652    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hltv\" (UniqueName: \"kubernetes.io/projected/07a2fd57-2c30-49d1-91f4-f461dde78747-kube-api-access-4hltv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-k86vt\" (UID: \"07a2fd57-2c30-49d1-91f4-f461dde78747\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:06.274949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.274866    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:06.686719 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.686688    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"]
Nov 01 08:38:06.688328 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:38:06.688300    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a2fd57_2c30_49d1_91f4_f461dde78747.slice/crio-8d954b3aed4ac3863c86592bf7c314e1b31e2fac676d0b2afc57f6e7736bfef5 WatchSource:0}: Error finding container 8d954b3aed4ac3863c86592bf7c314e1b31e2fac676d0b2afc57f6e7736bfef5: Status 404 returned error can't find the container with id 8d954b3aed4ac3863c86592bf7c314e1b31e2fac676d0b2afc57f6e7736bfef5
Nov 01 08:38:06.771613 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:06.771580    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt" event={"ID":"07a2fd57-2c30-49d1-91f4-f461dde78747","Type":"ContainerStarted","Data":"8d954b3aed4ac3863c86592bf7c314e1b31e2fac676d0b2afc57f6e7736bfef5"}
Nov 01 08:38:10.104783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.104749    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"]
Nov 01 08:38:10.115013 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.114990    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 08:38:10.165865 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.165833    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 01 08:38:10.166062 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.165916    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pcqjq\""
Nov 01 08:38:10.216049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.215945    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"]
Nov 01 08:38:10.234578 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.234537    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfl6b\" (UniqueName: \"kubernetes.io/projected/5cdcfa74-fd37-405c-8850-47dfa5b238e5-kube-api-access-vfl6b\") pod \"dns-operator-controller-manager-8557cbdb84-zslv4\" (UID: \"5cdcfa74-fd37-405c-8850-47dfa5b238e5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 08:38:10.335372 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.335324    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfl6b\" (UniqueName: \"kubernetes.io/projected/5cdcfa74-fd37-405c-8850-47dfa5b238e5-kube-api-access-vfl6b\") pod \"dns-operator-controller-manager-8557cbdb84-zslv4\" (UID: \"5cdcfa74-fd37-405c-8850-47dfa5b238e5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 08:38:10.403439 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.403353    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfl6b\" (UniqueName: \"kubernetes.io/projected/5cdcfa74-fd37-405c-8850-47dfa5b238e5-kube-api-access-vfl6b\") pod \"dns-operator-controller-manager-8557cbdb84-zslv4\" (UID: \"5cdcfa74-fd37-405c-8850-47dfa5b238e5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 08:38:10.431155 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.431120    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 08:38:10.815379 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:10.815349    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"]
Nov 01 08:38:10.818364 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:38:10.818333    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdcfa74_fd37_405c_8850_47dfa5b238e5.slice/crio-8c91dce2370c8a9328faf28a9e7e30652f96b5260adafccdff53c2293564e2e0 WatchSource:0}: Error finding container 8c91dce2370c8a9328faf28a9e7e30652f96b5260adafccdff53c2293564e2e0: Status 404 returned error can't find the container with id 8c91dce2370c8a9328faf28a9e7e30652f96b5260adafccdff53c2293564e2e0
Nov 01 08:38:11.798313 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:11.798272    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4" event={"ID":"5cdcfa74-fd37-405c-8850-47dfa5b238e5","Type":"ContainerStarted","Data":"8c91dce2370c8a9328faf28a9e7e30652f96b5260adafccdff53c2293564e2e0"}
Nov 01 08:38:13.809949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:13.809912    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt" event={"ID":"07a2fd57-2c30-49d1-91f4-f461dde78747","Type":"ContainerStarted","Data":"1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19"}
Nov 01 08:38:13.810376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:13.810014    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:14.023873 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:14.023812    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt" podStartSLOduration=2.8934842830000003 podStartE2EDuration="9.023797846s" totalImagesPullingTime="6.130313563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:13.97482355 +0000 UTC m=+364.089696515" watchObservedRunningTime="2026-11-01 08:38:14.023797846 +0000 UTC m=+364.138670811"
Nov 01 08:38:15.286405 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.286372    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"]
Nov 01 08:38:15.290217 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.290196    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"
Nov 01 08:38:15.344522 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.344494    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tnc8j\""
Nov 01 08:38:15.383678 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.383645    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8m24\" (UniqueName: \"kubernetes.io/projected/ea75c58e-9f4a-4401-bcea-548e683abe2e-kube-api-access-p8m24\") pod \"authorino-operator-7b7bb4bc77-6s9qr\" (UID: \"ea75c58e-9f4a-4401-bcea-548e683abe2e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"
Nov 01 08:38:15.393953 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.393927    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"]
Nov 01 08:38:15.484293 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.484261    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8m24\" (UniqueName: \"kubernetes.io/projected/ea75c58e-9f4a-4401-bcea-548e683abe2e-kube-api-access-p8m24\") pod \"authorino-operator-7b7bb4bc77-6s9qr\" (UID: \"ea75c58e-9f4a-4401-bcea-548e683abe2e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"
Nov 01 08:38:15.546645 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.546575    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8m24\" (UniqueName: \"kubernetes.io/projected/ea75c58e-9f4a-4401-bcea-548e683abe2e-kube-api-access-p8m24\") pod \"authorino-operator-7b7bb4bc77-6s9qr\" (UID: \"ea75c58e-9f4a-4401-bcea-548e683abe2e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"
Nov 01 08:38:15.604227 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.604194    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"
Nov 01 08:38:15.727946 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.727904    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"]
Nov 01 08:38:15.732504 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.732479    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"
Nov 01 08:38:15.782821 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.782790    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 01 08:38:15.782821 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.782818    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 01 08:38:15.783013 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.782835    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kdgf8\""
Nov 01 08:38:15.787335 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.787311    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwb65\" (UniqueName: \"kubernetes.io/projected/faa6c981-a887-4bec-b526-ae5b28a85ea9-kube-api-access-jwb65\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2\" (UID: \"faa6c981-a887-4bec-b526-ae5b28a85ea9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"
Nov 01 08:38:15.787435 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.787400    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6c981-a887-4bec-b526-ae5b28a85ea9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2\" (UID: \"faa6c981-a887-4bec-b526-ae5b28a85ea9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"
Nov 01 08:38:15.827398 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.827336    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"]
Nov 01 08:38:15.888038 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.888000    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6c981-a887-4bec-b526-ae5b28a85ea9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2\" (UID: \"faa6c981-a887-4bec-b526-ae5b28a85ea9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"
Nov 01 08:38:15.888237 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.888104    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwb65\" (UniqueName: \"kubernetes.io/projected/faa6c981-a887-4bec-b526-ae5b28a85ea9-kube-api-access-jwb65\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2\" (UID: \"faa6c981-a887-4bec-b526-ae5b28a85ea9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"
Nov 01 08:38:15.888372 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.888350    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6c981-a887-4bec-b526-ae5b28a85ea9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2\" (UID: \"faa6c981-a887-4bec-b526-ae5b28a85ea9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"
Nov 01 08:38:15.947125 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.947090    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwb65\" (UniqueName: \"kubernetes.io/projected/faa6c981-a887-4bec-b526-ae5b28a85ea9-kube-api-access-jwb65\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2\" (UID: \"faa6c981-a887-4bec-b526-ae5b28a85ea9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"
Nov 01 08:38:15.969725 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:15.969699    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"]
Nov 01 08:38:15.971227 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:38:15.971204    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea75c58e_9f4a_4401_bcea_548e683abe2e.slice/crio-9ea21b90134f19b0cb62ac9c0a5247ecdd796310d2be102e11481620396dee6f WatchSource:0}: Error finding container 9ea21b90134f19b0cb62ac9c0a5247ecdd796310d2be102e11481620396dee6f: Status 404 returned error can't find the container with id 9ea21b90134f19b0cb62ac9c0a5247ecdd796310d2be102e11481620396dee6f
Nov 01 08:38:16.043989 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:16.043957    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"
Nov 01 08:38:16.461410 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:16.461381    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2"]
Nov 01 08:38:16.462681 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:38:16.462644    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa6c981_a887_4bec_b526_ae5b28a85ea9.slice/crio-1ce4256203bccb42ec3308b0cf2bfdddd77ab55c9f3ef44b31a0ca592285ebf2 WatchSource:0}: Error finding container 1ce4256203bccb42ec3308b0cf2bfdddd77ab55c9f3ef44b31a0ca592285ebf2: Status 404 returned error can't find the container with id 1ce4256203bccb42ec3308b0cf2bfdddd77ab55c9f3ef44b31a0ca592285ebf2
Nov 01 08:38:16.828633 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:16.828549    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr" event={"ID":"ea75c58e-9f4a-4401-bcea-548e683abe2e","Type":"ContainerStarted","Data":"9ea21b90134f19b0cb62ac9c0a5247ecdd796310d2be102e11481620396dee6f"}
Nov 01 08:38:16.830048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:16.830004    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2" event={"ID":"faa6c981-a887-4bec-b526-ae5b28a85ea9","Type":"ContainerStarted","Data":"1ce4256203bccb42ec3308b0cf2bfdddd77ab55c9f3ef44b31a0ca592285ebf2"}
Nov 01 08:38:17.778223 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:17.778145    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-vtnv8"
Nov 01 08:38:17.836363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:17.836327    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4" event={"ID":"5cdcfa74-fd37-405c-8850-47dfa5b238e5","Type":"ContainerStarted","Data":"85a762d1dc056bc0832547d7f67e1dcff99a3a735eba705c0024e3c3476245ee"}
Nov 01 08:38:17.836507 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:17.836388    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 08:38:18.126873 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:18.126823    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4" podStartSLOduration=1.465423406 podStartE2EDuration="8.126806977s" totalImagesPullingTime="6.661383571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:18.125208235 +0000 UTC m=+368.240081203" watchObservedRunningTime="2026-11-01 08:38:18.126806977 +0000 UTC m=+368.241679944"
Nov 01 08:38:21.858866 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:21.858834    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr" event={"ID":"ea75c58e-9f4a-4401-bcea-548e683abe2e","Type":"ContainerStarted","Data":"2e7e35cd41723a348df155eb2a93ffe6718bc63d25ad3fa3ac945ef021732415"}
Nov 01 08:38:21.859289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:21.858940    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"
Nov 01 08:38:21.972348 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:21.972295    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr" podStartSLOduration=2.021871101 podStartE2EDuration="6.97228252s" totalImagesPullingTime="4.950411419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:21.971897075 +0000 UTC m=+372.086770055" watchObservedRunningTime="2026-11-01 08:38:21.97228252 +0000 UTC m=+372.087155488"
Nov 01 08:38:22.866101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:22.866064    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2" event={"ID":"faa6c981-a887-4bec-b526-ae5b28a85ea9","Type":"ContainerStarted","Data":"1443e996be06ff959e7642c40f093494aa4119dd5c9ebfe9f409b6e5c2d940ff"}
Nov 01 08:38:23.076083 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:23.075960    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4pzm2" podStartSLOduration=1.923749952 podStartE2EDuration="8.075942369s" totalImagesPullingTime="6.152192417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:23.027928376 +0000 UTC m=+373.142801344" watchObservedRunningTime="2026-11-01 08:38:23.075942369 +0000 UTC m=+373.190815335"
Nov 01 08:38:24.818865 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:24.818826    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:25.438935 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.438904    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"]
Nov 01 08:38:25.442498 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.442483    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:25.539718 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.539687    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"]
Nov 01 08:38:25.572439 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.572404    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thw78\" (UniqueName: \"kubernetes.io/projected/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-kube-api-access-thw78\") pod \"kuadrant-operator-controller-manager-678d87766f-vlfhp\" (UID: \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:25.572608 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.572526    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vlfhp\" (UID: \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:25.673629 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.673583    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vlfhp\" (UID: \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:25.673797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.673651    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thw78\" (UniqueName: \"kubernetes.io/projected/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-kube-api-access-thw78\") pod \"kuadrant-operator-controller-manager-678d87766f-vlfhp\" (UID: \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:25.674001 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.673983    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vlfhp\" (UID: \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:25.734698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.734617    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thw78\" (UniqueName: \"kubernetes.io/projected/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-kube-api-access-thw78\") pod \"kuadrant-operator-controller-manager-678d87766f-vlfhp\" (UID: \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:25.755619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:25.755587    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:26.204252 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:26.204217    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"]
Nov 01 08:38:26.205595 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:38:26.205570    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ef7b97_d4b7_4482_9026_98a0ffffeaf1.slice/crio-eb1cdbdaffc22dabc1e4c2bbeec7951406fec9a93fd2a6caf9c386050b47dde1 WatchSource:0}: Error finding container eb1cdbdaffc22dabc1e4c2bbeec7951406fec9a93fd2a6caf9c386050b47dde1: Status 404 returned error can't find the container with id eb1cdbdaffc22dabc1e4c2bbeec7951406fec9a93fd2a6caf9c386050b47dde1
Nov 01 08:38:26.886596 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:26.886563    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp" event={"ID":"76ef7b97-d4b7-4482-9026-98a0ffffeaf1","Type":"ContainerStarted","Data":"3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a"}
Nov 01 08:38:26.886807 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:26.886612    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp" event={"ID":"76ef7b97-d4b7-4482-9026-98a0ffffeaf1","Type":"ContainerStarted","Data":"eb1cdbdaffc22dabc1e4c2bbeec7951406fec9a93fd2a6caf9c386050b47dde1"}
Nov 01 08:38:26.886807 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:26.886729    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:27.104739 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:27.104686    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp" podStartSLOduration=2.104669337 podStartE2EDuration="2.104669337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:27.055772045 +0000 UTC m=+377.170645013" watchObservedRunningTime="2026-11-01 08:38:27.104669337 +0000 UTC m=+377.219542350"
Nov 01 08:38:28.843758 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:28.843722    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 08:38:31.023215 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:38:31.023176    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:38:32.869204 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:32.869170    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6s9qr"
Nov 01 08:38:33.735504 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.735465    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"]
Nov 01 08:38:33.742328 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.742304    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:33.794606 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.794572    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 08:38:33.794935 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.794917    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"]
Nov 01 08:38:33.840295 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.840263    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 08:38:33.845777 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.845740    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gfl7\" (UniqueName: \"kubernetes.io/projected/542cae9b-0322-4285-b0e0-3e399cc86bcb-kube-api-access-5gfl7\") pod \"cert-manager-webhook-5dbbbf75cc-79vq2\" (UID: \"542cae9b-0322-4285-b0e0-3e399cc86bcb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:33.845927 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.845797    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/542cae9b-0322-4285-b0e0-3e399cc86bcb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79vq2\" (UID: \"542cae9b-0322-4285-b0e0-3e399cc86bcb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:33.871968 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.871937    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7gmgl\""
Nov 01 08:38:33.947284 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.947252    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gfl7\" (UniqueName: \"kubernetes.io/projected/542cae9b-0322-4285-b0e0-3e399cc86bcb-kube-api-access-5gfl7\") pod \"cert-manager-webhook-5dbbbf75cc-79vq2\" (UID: \"542cae9b-0322-4285-b0e0-3e399cc86bcb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:33.947284 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.947292    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/542cae9b-0322-4285-b0e0-3e399cc86bcb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79vq2\" (UID: \"542cae9b-0322-4285-b0e0-3e399cc86bcb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:33.982996 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.982960    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"]
Nov 01 08:38:33.987121 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:33.987063    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.008446 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.008415    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/542cae9b-0322-4285-b0e0-3e399cc86bcb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79vq2\" (UID: \"542cae9b-0322-4285-b0e0-3e399cc86bcb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:34.008642 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.008617    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gfl7\" (UniqueName: \"kubernetes.io/projected/542cae9b-0322-4285-b0e0-3e399cc86bcb-kube-api-access-5gfl7\") pod \"cert-manager-webhook-5dbbbf75cc-79vq2\" (UID: \"542cae9b-0322-4285-b0e0-3e399cc86bcb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:34.048795 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.048761    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"]
Nov 01 08:38:34.065273 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.065242    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:34.099851 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.099827    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-db928\""
Nov 01 08:38:34.150616 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150579    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d1c892-bf07-467c-92db-655a8d317413-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.150760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150650    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.150760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150706    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.150760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150739    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.150861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150764    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5msmm\" (UniqueName: \"kubernetes.io/projected/75d1c892-bf07-467c-92db-655a8d317413-kube-api-access-5msmm\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.150861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150795    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d1c892-bf07-467c-92db-655a8d317413-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.150861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150858    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.150955 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150876    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d1c892-bf07-467c-92db-655a8d317413-istio-token\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.150955 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.150896    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-istio-data\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.251744 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.251659    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.251744 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.251700    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.251744 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.251719    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5msmm\" (UniqueName: \"kubernetes.io/projected/75d1c892-bf07-467c-92db-655a8d317413-kube-api-access-5msmm\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.251744 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.251745    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d1c892-bf07-467c-92db-655a8d317413-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.251947    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.251992    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d1c892-bf07-467c-92db-655a8d317413-istio-token\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.252054    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-istio-data\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252132 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.252115    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d1c892-bf07-467c-92db-655a8d317413-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252334 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.252162    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.252328    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252392 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.252362    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252555 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.252533    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.252786 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.252762    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-istio-data\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.253069 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.253020    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d1c892-bf07-467c-92db-655a8d317413-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.254213 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.254194    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d1c892-bf07-467c-92db-655a8d317413-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.254439 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.254423    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d1c892-bf07-467c-92db-655a8d317413-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.312388 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.312350    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d1c892-bf07-467c-92db-655a8d317413-istio-token\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.312531 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.312468    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5msmm\" (UniqueName: \"kubernetes.io/projected/75d1c892-bf07-467c-92db-655a8d317413-kube-api-access-5msmm\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fqdwtg\" (UID: \"75d1c892-bf07-467c-92db-655a8d317413\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.436916 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.436891    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"]
Nov 01 08:38:34.438219 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:38:34.438177    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod542cae9b_0322_4285_b0e0_3e399cc86bcb.slice/crio-07d9fd231e84cdc234da8cba254ef551891035df423518c8ca5db65fdd0304b9 WatchSource:0}: Error finding container 07d9fd231e84cdc234da8cba254ef551891035df423518c8ca5db65fdd0304b9: Status 404 returned error can't find the container with id 07d9fd231e84cdc234da8cba254ef551891035df423518c8ca5db65fdd0304b9
Nov 01 08:38:34.601198 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.601108    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:34.926634 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:34.926599    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2" event={"ID":"542cae9b-0322-4285-b0e0-3e399cc86bcb","Type":"ContainerStarted","Data":"07d9fd231e84cdc234da8cba254ef551891035df423518c8ca5db65fdd0304b9"}
Nov 01 08:38:35.023595 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:35.023564    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"]
Nov 01 08:38:35.025064 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:38:35.025005    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d1c892_bf07_467c_92db_655a8d317413.slice/crio-97f7a903c2d2273211883d8b99a996f17a30ac27bce08dfcc648c6af94469e5c WatchSource:0}: Error finding container 97f7a903c2d2273211883d8b99a996f17a30ac27bce08dfcc648c6af94469e5c: Status 404 returned error can't find the container with id 97f7a903c2d2273211883d8b99a996f17a30ac27bce08dfcc648c6af94469e5c
Nov 01 08:38:35.027220 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:35.027190    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:38:35.027329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:35.027252    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:38:35.027329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:35.027282    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:38:35.933737 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:35.933699    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg" event={"ID":"75d1c892-bf07-467c-92db-655a8d317413","Type":"ContainerStarted","Data":"753ab0883b81ebf53d7ab6d6051a06b7f35c44f2963e307c6d43e8890265579a"}
Nov 01 08:38:35.934176 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:35.933763    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg" event={"ID":"75d1c892-bf07-467c-92db-655a8d317413","Type":"ContainerStarted","Data":"97f7a903c2d2273211883d8b99a996f17a30ac27bce08dfcc648c6af94469e5c"}
Nov 01 08:38:36.100212 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:36.100163    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg" podStartSLOduration=3.100149055 podStartE2EDuration="3.100149055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:36.09851351 +0000 UTC m=+386.213386479" watchObservedRunningTime="2026-11-01 08:38:36.100149055 +0000 UTC m=+386.215022022"
Nov 01 08:38:36.602243 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:36.602210    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:36.608338 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:36.608304    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:36.939718 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:36.939636    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:36.940645 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:36.940629    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fqdwtg"
Nov 01 08:38:37.894378 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:37.894349    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 08:38:38.109716 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:38.109682    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"]
Nov 01 08:38:38.110501 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:38.109960    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt" podUID="07a2fd57-2c30-49d1-91f4-f461dde78747" containerName="manager" containerID="cri-o://1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19" gracePeriod=10
Nov 01 08:38:38.952226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:38.952133    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2" event={"ID":"542cae9b-0322-4285-b0e0-3e399cc86bcb","Type":"ContainerStarted","Data":"87a87e927ef97b760f1f85f3c775c8281368dc07b82aa47b610e576b7fb306ae"}
Nov 01 08:38:39.073389 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.073342    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2" podStartSLOduration=1.825062097 podStartE2EDuration="6.073327783s" totalImagesPullingTime="4.248265686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:39.071474095 +0000 UTC m=+389.186347063" watchObservedRunningTime="2026-11-01 08:38:39.073327783 +0000 UTC m=+389.188200750"
Nov 01 08:38:39.293893 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.293860    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:39.402778 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.402749    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a2fd57-2c30-49d1-91f4-f461dde78747-kube-api-access-4hltv\" (UniqueName: \"kubernetes.io/projected/07a2fd57-2c30-49d1-91f4-f461dde78747-kube-api-access-4hltv\") pod \"07a2fd57-2c30-49d1-91f4-f461dde78747\" (UID: \"07a2fd57-2c30-49d1-91f4-f461dde78747\") "
Nov 01 08:38:39.402986 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.402815    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a2fd57-2c30-49d1-91f4-f461dde78747-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a2fd57-2c30-49d1-91f4-f461dde78747-extensions-socket-volume\") pod \"07a2fd57-2c30-49d1-91f4-f461dde78747\" (UID: \"07a2fd57-2c30-49d1-91f4-f461dde78747\") "
Nov 01 08:38:39.403343 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.403316    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a2fd57-2c30-49d1-91f4-f461dde78747-extensions-socket-volume" pod "07a2fd57-2c30-49d1-91f4-f461dde78747" (UID: "07a2fd57-2c30-49d1-91f4-f461dde78747"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:38:39.405222 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.405201    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a2fd57-2c30-49d1-91f4-f461dde78747-kube-api-access-4hltv" pod "07a2fd57-2c30-49d1-91f4-f461dde78747" (UID: "07a2fd57-2c30-49d1-91f4-f461dde78747"). InnerVolumeSpecName "kube-api-access-4hltv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:38:39.503792 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.503757    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hltv\" (UniqueName: \"kubernetes.io/projected/07a2fd57-2c30-49d1-91f4-f461dde78747-kube-api-access-4hltv\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:38:39.503792 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.503791    4931 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07a2fd57-2c30-49d1-91f4-f461dde78747-extensions-socket-volume\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:38:39.958398 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.958366    4931 generic.go:356] "Generic (PLEG): container finished" podID="07a2fd57-2c30-49d1-91f4-f461dde78747" containerID="1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19" exitCode=0
Nov 01 08:38:39.958603 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.958415    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"
Nov 01 08:38:39.958603 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.958451    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt" event={"ID":"07a2fd57-2c30-49d1-91f4-f461dde78747","Type":"ContainerDied","Data":"1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19"}
Nov 01 08:38:39.958603 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.958510    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt" event={"ID":"07a2fd57-2c30-49d1-91f4-f461dde78747","Type":"ContainerDied","Data":"8d954b3aed4ac3863c86592bf7c314e1b31e2fac676d0b2afc57f6e7736bfef5"}
Nov 01 08:38:39.958603 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.958541    4931 scope.go:122] "RemoveContainer" containerID="1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19"
Nov 01 08:38:39.958824 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.958788    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:38:39.967999 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.967981    4931 scope.go:122] "RemoveContainer" containerID="1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19"
Nov 01 08:38:39.968316 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:38:39.968298    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19\": container with ID starting with 1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19 not found: ID does not exist" containerID="1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19"
Nov 01 08:38:39.968363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:39.968324    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19"} err="failed to get container status \"1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19\": rpc error: code = NotFound desc = could not find container \"1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19\": container with ID starting with 1fe0d32ebe83684c47fcca93f58aaa1ad0e36579426355e709eab037e9207d19 not found: ID does not exist"
Nov 01 08:38:40.168441 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.168406    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"]
Nov 01 08:38:40.217458 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.217376    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-k86vt"]
Nov 01 08:38:40.280481 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.280446    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb5q4"]
Nov 01 08:38:40.280858 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.280845    4931 cpu_manager.go:414] "containerMap: removing container" podUID="07a2fd57-2c30-49d1-91f4-f461dde78747" containerName="manager"
Nov 01 08:38:40.280911 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.280860    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a2fd57-2c30-49d1-91f4-f461dde78747" containerName="manager"
Nov 01 08:38:40.280942 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.280932    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a2fd57-2c30-49d1-91f4-f461dde78747" containerName="manager"
Nov 01 08:38:40.315470 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.315444    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fb5q4"
Nov 01 08:38:40.337216 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.337189    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb5q4"]
Nov 01 08:38:40.368528 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.368497    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wlrzh\""
Nov 01 08:38:40.512847 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.512760    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5w8g\" (UniqueName: \"kubernetes.io/projected/f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac-kube-api-access-m5w8g\") pod \"authorino-57f7f8659f-fb5q4\" (UID: \"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac\") " pod="kuadrant-system/authorino-57f7f8659f-fb5q4"
Nov 01 08:38:40.613976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.613935    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5w8g\" (UniqueName: \"kubernetes.io/projected/f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac-kube-api-access-m5w8g\") pod \"authorino-57f7f8659f-fb5q4\" (UID: \"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac\") " pod="kuadrant-system/authorino-57f7f8659f-fb5q4"
Nov 01 08:38:40.672808 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.672777    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5w8g\" (UniqueName: \"kubernetes.io/projected/f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac-kube-api-access-m5w8g\") pod \"authorino-57f7f8659f-fb5q4\" (UID: \"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac\") " pod="kuadrant-system/authorino-57f7f8659f-fb5q4"
Nov 01 08:38:40.943197 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:40.943157    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fb5q4"
Nov 01 08:38:41.329339 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:41.329306    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb5q4"]
Nov 01 08:38:41.330744 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:38:41.330716    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3af4aeb_8c1d_41d5_9c3d_9f320334e1ac.slice/crio-4caba2e4408a70c5f472144c65ef68e56a23902ba919bfdb437dd9242f65fafd WatchSource:0}: Error finding container 4caba2e4408a70c5f472144c65ef68e56a23902ba919bfdb437dd9242f65fafd: Status 404 returned error can't find the container with id 4caba2e4408a70c5f472144c65ef68e56a23902ba919bfdb437dd9242f65fafd
Nov 01 08:38:41.971154 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:41.971117    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fb5q4" event={"ID":"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac","Type":"ContainerStarted","Data":"4caba2e4408a70c5f472144c65ef68e56a23902ba919bfdb437dd9242f65fafd"}
Nov 01 08:38:42.046779 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:42.046746    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a2fd57-2c30-49d1-91f4-f461dde78747" path="/var/lib/kubelet/pods/07a2fd57-2c30-49d1-91f4-f461dde78747/volumes"
Nov 01 08:38:44.988693 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:44.988648    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fb5q4" event={"ID":"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac","Type":"ContainerStarted","Data":"a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4"}
Nov 01 08:38:45.099201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:45.099144    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-fb5q4" podStartSLOduration=2.262139299 podStartE2EDuration="5.099128538s" totalImagesPullingTime="2.836989239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:45.097422697 +0000 UTC m=+395.212295665" watchObservedRunningTime="2026-11-01 08:38:45.099128538 +0000 UTC m=+395.214001505"
Nov 01 08:38:45.968451 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:38:45.968417    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79vq2"
Nov 01 08:39:00.061580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.061505    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"]
Nov 01 08:39:00.065035 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.065001    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:00.119254 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.119222    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"]
Nov 01 08:39:00.164486 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.164455    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wx6gn\""
Nov 01 08:39:00.164486 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.164472    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 08:39:00.185187 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.185147    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwt2d\" (UniqueName: \"kubernetes.io/projected/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-kube-api-access-bwt2d\") pod \"limitador-limitador-74dfcfb894-9dzcb\" (UID: \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:00.185393 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.185311    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-config-file\") pod \"limitador-limitador-74dfcfb894-9dzcb\" (UID: \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:00.286458 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.286418    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-config-file\") pod \"limitador-limitador-74dfcfb894-9dzcb\" (UID: \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:00.286666 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.286473    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwt2d\" (UniqueName: \"kubernetes.io/projected/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-kube-api-access-bwt2d\") pod \"limitador-limitador-74dfcfb894-9dzcb\" (UID: \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:00.287166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.287141    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-config-file\") pod \"limitador-limitador-74dfcfb894-9dzcb\" (UID: \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:00.301497 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.301464    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"]
Nov 01 08:39:00.345675 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.345602    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwt2d\" (UniqueName: \"kubernetes.io/projected/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-kube-api-access-bwt2d\") pod \"limitador-limitador-74dfcfb894-9dzcb\" (UID: \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:00.377417 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.377380    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:00.745664 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:00.745642    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"]
Nov 01 08:39:00.747213 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:39:00.747185    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e480bfb_ccda_49ec_bc76_b350e0ba1c16.slice/crio-6d7ee7d49c5e7011a018d6e2af742d510743aeb1dc804c268d831b94719d0462 WatchSource:0}: Error finding container 6d7ee7d49c5e7011a018d6e2af742d510743aeb1dc804c268d831b94719d0462: Status 404 returned error can't find the container with id 6d7ee7d49c5e7011a018d6e2af742d510743aeb1dc804c268d831b94719d0462
Nov 01 08:39:01.021960 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:39:01.021873    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:39:01.068135 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:01.068098    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb" event={"ID":"8e480bfb-ccda-49ec-bc76-b350e0ba1c16","Type":"ContainerStarted","Data":"6d7ee7d49c5e7011a018d6e2af742d510743aeb1dc804c268d831b94719d0462"}
Nov 01 08:39:05.736650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:05.736616    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb5q4"]
Nov 01 08:39:05.737097 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:05.736794    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-fb5q4" podUID="f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac" containerName="authorino" containerID="cri-o://a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4" gracePeriod=30
Nov 01 08:39:05.997559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:05.997497    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fb5q4"
Nov 01 08:39:06.094330 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.094299    4931 generic.go:356] "Generic (PLEG): container finished" podID="f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac" containerID="a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4" exitCode=0
Nov 01 08:39:06.094498 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.094347    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fb5q4"
Nov 01 08:39:06.094498 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.094392    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fb5q4" event={"ID":"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac","Type":"ContainerDied","Data":"a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4"}
Nov 01 08:39:06.094498 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.094439    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fb5q4" event={"ID":"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac","Type":"ContainerDied","Data":"4caba2e4408a70c5f472144c65ef68e56a23902ba919bfdb437dd9242f65fafd"}
Nov 01 08:39:06.094498 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.094456    4931 scope.go:122] "RemoveContainer" containerID="a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4"
Nov 01 08:39:06.102789 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.102772    4931 scope.go:122] "RemoveContainer" containerID="a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4"
Nov 01 08:39:06.103090 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:39:06.103063    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4\": container with ID starting with a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4 not found: ID does not exist" containerID="a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4"
Nov 01 08:39:06.103192 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.103101    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4"} err="failed to get container status \"a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4\": rpc error: code = NotFound desc = could not find container \"a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4\": container with ID starting with a6b7f1c77ad523d2b15b2fa1ed0d80169d75df05df077214ea305f4e13b670f4 not found: ID does not exist"
Nov 01 08:39:06.138809 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.138776    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac-kube-api-access-m5w8g\" (UniqueName: \"kubernetes.io/projected/f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac-kube-api-access-m5w8g\") pod \"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac\" (UID: \"f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac\") "
Nov 01 08:39:06.140961 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.140932    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac-kube-api-access-m5w8g" pod "f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac" (UID: "f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac"). InnerVolumeSpecName "kube-api-access-m5w8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:39:06.240077 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.240013    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5w8g\" (UniqueName: \"kubernetes.io/projected/f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac-kube-api-access-m5w8g\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:39:06.512194 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.512155    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb5q4"]
Nov 01 08:39:06.516089 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:06.516065    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fb5q4"]
Nov 01 08:39:08.046302 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:08.046269    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac" path="/var/lib/kubelet/pods/f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac/volumes"
Nov 01 08:39:16.153901 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:16.153864    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb" event={"ID":"8e480bfb-ccda-49ec-bc76-b350e0ba1c16","Type":"ContainerStarted","Data":"a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95"}
Nov 01 08:39:16.154360 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:16.153956    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:16.266482 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:16.266427    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb" podStartSLOduration=1.7756655829999999 podStartE2EDuration="16.26641229s" totalImagesPullingTime="14.490746707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:16.264765956 +0000 UTC m=+426.379638925" watchObservedRunningTime="2026-11-01 08:39:16.26641229 +0000 UTC m=+426.381285260"
Nov 01 08:39:18.760348 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:18.760314    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"]
Nov 01 08:39:18.760776 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:18.760516    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb" podUID="8e480bfb-ccda-49ec-bc76-b350e0ba1c16" containerName="limitador" containerID="cri-o://a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95" gracePeriod=30
Nov 01 08:39:19.311500 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:19.311476    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:19.459457 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:19.459421    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-config-file\" (UniqueName: \"kubernetes.io/configmap/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-config-file\") pod \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\" (UID: \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\") "
Nov 01 08:39:19.459631 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:19.459527    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-kube-api-access-bwt2d\" (UniqueName: \"kubernetes.io/projected/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-kube-api-access-bwt2d\") pod \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\" (UID: \"8e480bfb-ccda-49ec-bc76-b350e0ba1c16\") "
Nov 01 08:39:19.459776 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:19.459754    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-config-file" pod "8e480bfb-ccda-49ec-bc76-b350e0ba1c16" (UID: "8e480bfb-ccda-49ec-bc76-b350e0ba1c16"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:39:19.461785 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:19.461753    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-kube-api-access-bwt2d" pod "8e480bfb-ccda-49ec-bc76-b350e0ba1c16" (UID: "8e480bfb-ccda-49ec-bc76-b350e0ba1c16"). InnerVolumeSpecName "kube-api-access-bwt2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:39:19.560648 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:19.560612    4931 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-config-file\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:39:19.560648 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:19.560641    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwt2d\" (UniqueName: \"kubernetes.io/projected/8e480bfb-ccda-49ec-bc76-b350e0ba1c16-kube-api-access-bwt2d\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:39:20.177390 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.177351    4931 generic.go:356] "Generic (PLEG): container finished" podID="8e480bfb-ccda-49ec-bc76-b350e0ba1c16" containerID="a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95" exitCode=0
Nov 01 08:39:20.177871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.177397    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb" event={"ID":"8e480bfb-ccda-49ec-bc76-b350e0ba1c16","Type":"ContainerDied","Data":"a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95"}
Nov 01 08:39:20.177871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.177440    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb" event={"ID":"8e480bfb-ccda-49ec-bc76-b350e0ba1c16","Type":"ContainerDied","Data":"6d7ee7d49c5e7011a018d6e2af742d510743aeb1dc804c268d831b94719d0462"}
Nov 01 08:39:20.177871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.177461    4931 scope.go:122] "RemoveContainer" containerID="a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95"
Nov 01 08:39:20.177871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.177398    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"
Nov 01 08:39:20.186522 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.186502    4931 scope.go:122] "RemoveContainer" containerID="a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95"
Nov 01 08:39:20.186802 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:39:20.186781    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95\": container with ID starting with a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95 not found: ID does not exist" containerID="a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95"
Nov 01 08:39:20.186843 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.186812    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95"} err="failed to get container status \"a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95\": rpc error: code = NotFound desc = could not find container \"a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95\": container with ID starting with a66196bbceb3665ce6198768105c6a8ff0310c4036c304aebf9335ff19e25c95 not found: ID does not exist"
Nov 01 08:39:20.334593 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.334509    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"]
Nov 01 08:39:20.336813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:20.336795    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9dzcb"]
Nov 01 08:39:22.046397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:22.046362    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e480bfb-ccda-49ec-bc76-b350e0ba1c16" path="/var/lib/kubelet/pods/8e480bfb-ccda-49ec-bc76-b350e0ba1c16/volumes"
Nov 01 08:39:25.432963 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.432930    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"]
Nov 01 08:39:25.433376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.433297    4931 cpu_manager.go:414] "containerMap: removing container" podUID="f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac" containerName="authorino"
Nov 01 08:39:25.433376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.433308    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac" containerName="authorino"
Nov 01 08:39:25.433376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.433327    4931 cpu_manager.go:414] "containerMap: removing container" podUID="8e480bfb-ccda-49ec-bc76-b350e0ba1c16" containerName="limitador"
Nov 01 08:39:25.433376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.433332    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e480bfb-ccda-49ec-bc76-b350e0ba1c16" containerName="limitador"
Nov 01 08:39:25.433507 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.433391    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3af4aeb-8c1d-41d5-9c3d-9f320334e1ac" containerName="authorino"
Nov 01 08:39:25.433507 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.433401    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e480bfb-ccda-49ec-bc76-b350e0ba1c16" containerName="limitador"
Nov 01 08:39:25.437795 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.437778    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:25.490422 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.490393    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"]
Nov 01 08:39:25.505771 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.505747    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 08:39:25.535900 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.535871    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 01 08:39:25.535900 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.535871    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 08:39:25.536123 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.535997    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jqsx8\""
Nov 01 08:39:25.615588 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.615553    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/689189ee-73d6-41dc-920f-5f3108439b62-data\") pod \"postgres-7cdbf6dc44-bpdpw\" (UID: \"689189ee-73d6-41dc-920f-5f3108439b62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:25.615789 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.615607    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnbp\" (UniqueName: \"kubernetes.io/projected/689189ee-73d6-41dc-920f-5f3108439b62-kube-api-access-7mnbp\") pod \"postgres-7cdbf6dc44-bpdpw\" (UID: \"689189ee-73d6-41dc-920f-5f3108439b62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:25.716555 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.716474    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnbp\" (UniqueName: \"kubernetes.io/projected/689189ee-73d6-41dc-920f-5f3108439b62-kube-api-access-7mnbp\") pod \"postgres-7cdbf6dc44-bpdpw\" (UID: \"689189ee-73d6-41dc-920f-5f3108439b62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:25.716555 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.716539    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/689189ee-73d6-41dc-920f-5f3108439b62-data\") pod \"postgres-7cdbf6dc44-bpdpw\" (UID: \"689189ee-73d6-41dc-920f-5f3108439b62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:25.716966 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.716944    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/689189ee-73d6-41dc-920f-5f3108439b62-data\") pod \"postgres-7cdbf6dc44-bpdpw\" (UID: \"689189ee-73d6-41dc-920f-5f3108439b62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:25.775872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:25.775845    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnbp\" (UniqueName: \"kubernetes.io/projected/689189ee-73d6-41dc-920f-5f3108439b62-kube-api-access-7mnbp\") pod \"postgres-7cdbf6dc44-bpdpw\" (UID: \"689189ee-73d6-41dc-920f-5f3108439b62\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:26.050471 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:26.050397    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:26.416018 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:26.415993    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"]
Nov 01 08:39:26.417977 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:39:26.417946    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689189ee_73d6_41dc_920f_5f3108439b62.slice/crio-d0bda247b4144499769fe551315fd621e60691ef9268286ac7080aa33457c7c9 WatchSource:0}: Error finding container d0bda247b4144499769fe551315fd621e60691ef9268286ac7080aa33457c7c9: Status 404 returned error can't find the container with id d0bda247b4144499769fe551315fd621e60691ef9268286ac7080aa33457c7c9
Nov 01 08:39:27.214835 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:27.214798    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw" event={"ID":"689189ee-73d6-41dc-920f-5f3108439b62","Type":"ContainerStarted","Data":"d0bda247b4144499769fe551315fd621e60691ef9268286ac7080aa33457c7c9"}
Nov 01 08:39:31.028583 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:39:31.028530    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:39:34.256408 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:34.256375    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw" event={"ID":"689189ee-73d6-41dc-920f-5f3108439b62","Type":"ContainerStarted","Data":"fa00247a6ee7a0bd74e2b3b033b48a03e861e30c8536ae6d00172a8d3f56e95c"}
Nov 01 08:39:34.256817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:34.256556    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:34.368332 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:34.368283    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw" podStartSLOduration=1.9444158040000001 podStartE2EDuration="9.368269617s" totalImagesPullingTime="7.423853813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:34.366583573 +0000 UTC m=+444.481456540" watchObservedRunningTime="2026-11-01 08:39:34.368269617 +0000 UTC m=+444.483142584"
Nov 01 08:39:40.290746 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:40.290718    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bpdpw"
Nov 01 08:39:47.277629 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.277598    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fb4fcc885-r994f"]
Nov 01 08:39:47.303860 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.303828    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.354111 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.354053    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 08:39:47.396249 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.396215    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blm2l\" (UniqueName: \"kubernetes.io/projected/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-kube-api-access-blm2l\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.396412 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.396267    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-metrics-certs\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.396412 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.396350    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-webhook-cert\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.401604 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.401569    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 08:39:47.401604 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.401601    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ld8j5\""
Nov 01 08:39:47.401844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.401637    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 08:39:47.401844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.401662    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 08:39:47.401844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.401679    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 08:39:47.403796 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.403774    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fb4fcc885-r994f"]
Nov 01 08:39:47.497385 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.497346    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blm2l\" (UniqueName: \"kubernetes.io/projected/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-kube-api-access-blm2l\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.497565 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.497401    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-metrics-certs\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.497565 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.497438    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-webhook-cert\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.500097 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.500077    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-webhook-cert\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.500185 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.500169    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-metrics-certs\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.556829 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.556759    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blm2l\" (UniqueName: \"kubernetes.io/projected/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-kube-api-access-blm2l\") pod \"maas-controller-6fb4fcc885-r994f\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") " pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.617935 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.617901    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:47.842827 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.842744    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-d6hgb"]
Nov 01 08:39:47.847492 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.847471    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:47.900852 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.900814    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-webhook-cert\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:47.901047 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.900933    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-metrics-certs\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:47.901047 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.900958    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhlg\" (UniqueName: \"kubernetes.io/projected/785d692c-6c25-4f84-874a-136d9e082df1-kube-api-access-twhlg\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:47.940951 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:47.940919    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-d6hgb"]
Nov 01 08:39:48.000943 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.000909    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fb4fcc885-r994f"]
Nov 01 08:39:48.002420 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.002393    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-webhook-cert\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:48.002541 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.002454    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-metrics-certs\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:48.002541 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.002475    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhlg\" (UniqueName: \"kubernetes.io/projected/785d692c-6c25-4f84-874a-136d9e082df1-kube-api-access-twhlg\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:48.005084 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.005063    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-webhook-cert\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:48.005189 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.005129    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-metrics-certs\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:48.034146 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:39:48.034116    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe83fc60_2039_4d54_a1f3_8f3e5351d9d5.slice/crio-cef0a567a0b4b4fdb70b5f7208782c08b140a8f41014adeb3f22189fa2638385 WatchSource:0}: Error finding container cef0a567a0b4b4fdb70b5f7208782c08b140a8f41014adeb3f22189fa2638385: Status 404 returned error can't find the container with id cef0a567a0b4b4fdb70b5f7208782c08b140a8f41014adeb3f22189fa2638385
Nov 01 08:39:48.061670 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.061636    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhlg\" (UniqueName: \"kubernetes.io/projected/785d692c-6c25-4f84-874a-136d9e082df1-kube-api-access-twhlg\") pod \"maas-controller-7d89f6bbc8-d6hgb\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") " pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:48.161151 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.161103    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:48.323118 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.323083    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fb4fcc885-r994f" event={"ID":"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5","Type":"ContainerStarted","Data":"cef0a567a0b4b4fdb70b5f7208782c08b140a8f41014adeb3f22189fa2638385"}
Nov 01 08:39:48.530232 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:48.530204    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-d6hgb"]
Nov 01 08:39:48.531724 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:39:48.531696    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785d692c_6c25_4f84_874a_136d9e082df1.slice/crio-a8a87adf7f7cbfaa4f4f109bd5eed9c773a04cd941a0d2a1c8c267ec58c648b5 WatchSource:0}: Error finding container a8a87adf7f7cbfaa4f4f109bd5eed9c773a04cd941a0d2a1c8c267ec58c648b5: Status 404 returned error can't find the container with id a8a87adf7f7cbfaa4f4f109bd5eed9c773a04cd941a0d2a1c8c267ec58c648b5
Nov 01 08:39:49.329186 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:49.329149    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb" event={"ID":"785d692c-6c25-4f84-874a-136d9e082df1","Type":"ContainerStarted","Data":"a8a87adf7f7cbfaa4f4f109bd5eed9c773a04cd941a0d2a1c8c267ec58c648b5"}
Nov 01 08:39:57.374785 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:57.374741    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fb4fcc885-r994f" event={"ID":"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5","Type":"ContainerStarted","Data":"e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2"}
Nov 01 08:39:57.375315 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:57.374803    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:39:57.376311 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:57.376289    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb" event={"ID":"785d692c-6c25-4f84-874a-136d9e082df1","Type":"ContainerStarted","Data":"9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6"}
Nov 01 08:39:57.376441 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:57.376337    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:39:57.489125 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:57.489074    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6fb4fcc885-r994f" podStartSLOduration=2.068294249 podStartE2EDuration="10.489059472s" totalImagesPullingTime="8.420765223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:57.487836362 +0000 UTC m=+467.602709342" watchObservedRunningTime="2026-11-01 08:39:57.489059472 +0000 UTC m=+467.603932439"
Nov 01 08:39:57.604541 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:39:57.604489    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb" podStartSLOduration=2.672305619 podStartE2EDuration="10.604475129s" totalImagesPullingTime="7.93216951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:57.603064509 +0000 UTC m=+467.717937476" watchObservedRunningTime="2026-11-01 08:39:57.604475129 +0000 UTC m=+467.719348097"
Nov 01 08:40:01.023925 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:40:01.023890    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:40:01.495281 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.495250    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"]
Nov 01 08:40:01.499379 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.499354    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.551625 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.551595    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 01 08:40:01.551625 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.551598    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 01 08:40:01.551625 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.551598    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 01 08:40:01.599444 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.599416    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-jcspj\""
Nov 01 08:40:01.602872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.602850    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"]
Nov 01 08:40:01.624816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.624780    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq87j\" (UniqueName: \"kubernetes.io/projected/f5f45383-80fa-4ee1-9556-96ba0b54a988-kube-api-access-hq87j\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.625014 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.624857    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.625014 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.624946    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.625137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.625122    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5f45383-80fa-4ee1-9556-96ba0b54a988-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.726131 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.726097    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.726314 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.726153    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.726314 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:40:01.726224    4931 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 01 08:40:01.726439 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.726238    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5f45383-80fa-4ee1-9556-96ba0b54a988-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.726439 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:40:01.726335    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-proxy-source podName:f5f45383-80fa-4ee1-9556-96ba0b54a988 nodeName:}" failed. No retries permitted until 2026-11-01 08:40:02.226310885 +0000 UTC m=+472.341183844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-dcm8b" (UID: "f5f45383-80fa-4ee1-9556-96ba0b54a988") : configmap "usage-tenancy-proxy-source" not found
Nov 01 08:40:01.726439 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.726392    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq87j\" (UniqueName: \"kubernetes.io/projected/f5f45383-80fa-4ee1-9556-96ba0b54a988-kube-api-access-hq87j\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.726870 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.726852    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.728762 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.728741    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5f45383-80fa-4ee1-9556-96ba0b54a988-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:01.785371 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:01.785298    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq87j\" (UniqueName: \"kubernetes.io/projected/f5f45383-80fa-4ee1-9556-96ba0b54a988-kube-api-access-hq87j\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:02.231602 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:02.231570    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:02.232378 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:02.232358    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f5f45383-80fa-4ee1-9556-96ba0b54a988-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-dcm8b\" (UID: \"f5f45383-80fa-4ee1-9556-96ba0b54a988\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:02.414146 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:02.414113    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:02.783469 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:02.783444    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"]
Nov 01 08:40:02.785205 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:40:02.785162    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f45383_80fa_4ee1_9556_96ba0b54a988.slice/crio-dd087f494ef248f0dc9700e9a6e91d000b5f2fedcafb6e03bec53e878e1f5442 WatchSource:0}: Error finding container dd087f494ef248f0dc9700e9a6e91d000b5f2fedcafb6e03bec53e878e1f5442: Status 404 returned error can't find the container with id dd087f494ef248f0dc9700e9a6e91d000b5f2fedcafb6e03bec53e878e1f5442
Nov 01 08:40:03.407734 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.407695    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b" event={"ID":"f5f45383-80fa-4ee1-9556-96ba0b54a988","Type":"ContainerStarted","Data":"dd087f494ef248f0dc9700e9a6e91d000b5f2fedcafb6e03bec53e878e1f5442"}
Nov 01 08:40:03.495200 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.495162    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-z247t"]
Nov 01 08:40:03.500148 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.500128    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:03.543398 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.543366    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0775c9-babb-41b8-8db2-f958a8f69c95-cert\") pod \"kserve-controller-manager-776cc66d47-z247t\" (UID: \"cc0775c9-babb-41b8-8db2-f958a8f69c95\") " pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:03.543398 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.543403    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hsb4\" (UniqueName: \"kubernetes.io/projected/cc0775c9-babb-41b8-8db2-f958a8f69c95-kube-api-access-4hsb4\") pod \"kserve-controller-manager-776cc66d47-z247t\" (UID: \"cc0775c9-babb-41b8-8db2-f958a8f69c95\") " pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:03.550892 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.550861    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9tvr8\""
Nov 01 08:40:03.551645 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.551501    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 01 08:40:03.553384 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.553362    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-z247t"]
Nov 01 08:40:03.600461 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.600431    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"]
Nov 01 08:40:03.604038 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.604008    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:03.644136 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.644103    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e013612f-52d5-4e9a-9e1e-5090e63a6fde-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8gdpd\" (UID: \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:03.644327 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.644150    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0775c9-babb-41b8-8db2-f958a8f69c95-cert\") pod \"kserve-controller-manager-776cc66d47-z247t\" (UID: \"cc0775c9-babb-41b8-8db2-f958a8f69c95\") " pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:03.644327 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.644182    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hsb4\" (UniqueName: \"kubernetes.io/projected/cc0775c9-babb-41b8-8db2-f958a8f69c95-kube-api-access-4hsb4\") pod \"kserve-controller-manager-776cc66d47-z247t\" (UID: \"cc0775c9-babb-41b8-8db2-f958a8f69c95\") " pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:03.644327 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.644301    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92wv\" (UniqueName: \"kubernetes.io/projected/e013612f-52d5-4e9a-9e1e-5090e63a6fde-kube-api-access-d92wv\") pod \"llmisvc-controller-manager-869d5dcc5d-8gdpd\" (UID: \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:03.646736 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.646713    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0775c9-babb-41b8-8db2-f958a8f69c95-cert\") pod \"kserve-controller-manager-776cc66d47-z247t\" (UID: \"cc0775c9-babb-41b8-8db2-f958a8f69c95\") " pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:03.655010 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.654984    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-k8s7j\""
Nov 01 08:40:03.657626 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.657597    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"]
Nov 01 08:40:03.705589 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.705522    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hsb4\" (UniqueName: \"kubernetes.io/projected/cc0775c9-babb-41b8-8db2-f958a8f69c95-kube-api-access-4hsb4\") pod \"kserve-controller-manager-776cc66d47-z247t\" (UID: \"cc0775c9-babb-41b8-8db2-f958a8f69c95\") " pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:03.706997 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.706974    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dssdt"]
Nov 01 08:40:03.710750 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.710727    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:03.745351 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.745310    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e013612f-52d5-4e9a-9e1e-5090e63a6fde-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8gdpd\" (UID: \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:03.745516 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.745422    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92wv\" (UniqueName: \"kubernetes.io/projected/e013612f-52d5-4e9a-9e1e-5090e63a6fde-kube-api-access-d92wv\") pod \"llmisvc-controller-manager-869d5dcc5d-8gdpd\" (UID: \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:03.752116 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.752099    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 01 08:40:03.758573 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.758553    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e013612f-52d5-4e9a-9e1e-5090e63a6fde-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8gdpd\" (UID: \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:03.765612 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.765582    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dssdt"]
Nov 01 08:40:03.806006 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.805970    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92wv\" (UniqueName: \"kubernetes.io/projected/e013612f-52d5-4e9a-9e1e-5090e63a6fde-kube-api-access-d92wv\") pod \"llmisvc-controller-manager-869d5dcc5d-8gdpd\" (UID: \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:03.817916 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.817887    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:03.846400 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.846329    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkq5\" (UniqueName: \"kubernetes.io/projected/8d282f7e-e082-4cc0-ac9b-39880957b56e-kube-api-access-7pkq5\") pod \"model-serving-api-75bd89fdfc-dssdt\" (UID: \"8d282f7e-e082-4cc0-ac9b-39880957b56e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:03.846573 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.846448    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d282f7e-e082-4cc0-ac9b-39880957b56e-tls-certs\") pod \"model-serving-api-75bd89fdfc-dssdt\" (UID: \"8d282f7e-e082-4cc0-ac9b-39880957b56e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:03.852271 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.852250    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-l6tqt\""
Nov 01 08:40:03.852388 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.852367    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 01 08:40:03.917357 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.917328    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:03.947812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.947776    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkq5\" (UniqueName: \"kubernetes.io/projected/8d282f7e-e082-4cc0-ac9b-39880957b56e-kube-api-access-7pkq5\") pod \"model-serving-api-75bd89fdfc-dssdt\" (UID: \"8d282f7e-e082-4cc0-ac9b-39880957b56e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:03.947966 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.947863    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d282f7e-e082-4cc0-ac9b-39880957b56e-tls-certs\") pod \"model-serving-api-75bd89fdfc-dssdt\" (UID: \"8d282f7e-e082-4cc0-ac9b-39880957b56e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:03.950390 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:03.950368    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d282f7e-e082-4cc0-ac9b-39880957b56e-tls-certs\") pod \"model-serving-api-75bd89fdfc-dssdt\" (UID: \"8d282f7e-e082-4cc0-ac9b-39880957b56e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:04.007667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:04.007629    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkq5\" (UniqueName: \"kubernetes.io/projected/8d282f7e-e082-4cc0-ac9b-39880957b56e-kube-api-access-7pkq5\") pod \"model-serving-api-75bd89fdfc-dssdt\" (UID: \"8d282f7e-e082-4cc0-ac9b-39880957b56e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:04.026550 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:04.026516    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:04.184215 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:04.184183    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-z247t"]
Nov 01 08:40:04.184897 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:40:04.184871    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0775c9_babb_41b8_8db2_f958a8f69c95.slice/crio-d61501723d2b7fcd71b9921a23cd53565cc4547ae7a5309b5a261231ad586fbf WatchSource:0}: Error finding container d61501723d2b7fcd71b9921a23cd53565cc4547ae7a5309b5a261231ad586fbf: Status 404 returned error can't find the container with id d61501723d2b7fcd71b9921a23cd53565cc4547ae7a5309b5a261231ad586fbf
Nov 01 08:40:04.285443 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:04.285421    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"]
Nov 01 08:40:04.286810 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:40:04.286787    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode013612f_52d5_4e9a_9e1e_5090e63a6fde.slice/crio-5a977cfeb4682b92b722f34b47a5cb9c43d81ea0664c3c826d4c0c88e6c62d0f WatchSource:0}: Error finding container 5a977cfeb4682b92b722f34b47a5cb9c43d81ea0664c3c826d4c0c88e6c62d0f: Status 404 returned error can't find the container with id 5a977cfeb4682b92b722f34b47a5cb9c43d81ea0664c3c826d4c0c88e6c62d0f
Nov 01 08:40:04.395372 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:04.395326    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dssdt"]
Nov 01 08:40:04.405426 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:40:04.405394    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d282f7e_e082_4cc0_ac9b_39880957b56e.slice/crio-4c0afb4f4fdff35261dc184a4fee62d2d3fb7424ffbb0790b7e8778b29379c7d WatchSource:0}: Error finding container 4c0afb4f4fdff35261dc184a4fee62d2d3fb7424ffbb0790b7e8778b29379c7d: Status 404 returned error can't find the container with id 4c0afb4f4fdff35261dc184a4fee62d2d3fb7424ffbb0790b7e8778b29379c7d
Nov 01 08:40:04.414794 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:04.414756    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-dssdt" event={"ID":"8d282f7e-e082-4cc0-ac9b-39880957b56e","Type":"ContainerStarted","Data":"4c0afb4f4fdff35261dc184a4fee62d2d3fb7424ffbb0790b7e8778b29379c7d"}
Nov 01 08:40:04.416064 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:04.416020    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd" event={"ID":"e013612f-52d5-4e9a-9e1e-5090e63a6fde","Type":"ContainerStarted","Data":"5a977cfeb4682b92b722f34b47a5cb9c43d81ea0664c3c826d4c0c88e6c62d0f"}
Nov 01 08:40:04.417194 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:04.417173    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-z247t" event={"ID":"cc0775c9-babb-41b8-8db2-f958a8f69c95","Type":"ContainerStarted","Data":"d61501723d2b7fcd71b9921a23cd53565cc4547ae7a5309b5a261231ad586fbf"}
Nov 01 08:40:08.390885 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:08.389942    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:40:08.392242 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:08.391615    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 08:40:08.653635 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:08.652360    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fb4fcc885-r994f"]
Nov 01 08:40:08.653635 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:08.652648    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6fb4fcc885-r994f" podUID="fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" containerName="manager" containerID="cri-o://e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2" gracePeriod=30
Nov 01 08:40:08.970767 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:08.970737    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:40:09.102979 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.102921    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-kube-api-access-blm2l\" (UniqueName: \"kubernetes.io/projected/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-kube-api-access-blm2l\") pod \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") "
Nov 01 08:40:09.103175 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.103057    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-webhook-cert\") pod \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") "
Nov 01 08:40:09.103175 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.103143    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-metrics-certs\") pod \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\" (UID: \"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5\") "
Nov 01 08:40:09.106907 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.106865    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-metrics-certs" pod "fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" (UID: "fe83fc60-2039-4d54-a1f3-8f3e5351d9d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:40:09.117631 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.117575    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-webhook-cert" pod "fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" (UID: "fe83fc60-2039-4d54-a1f3-8f3e5351d9d5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:40:09.118300 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.118269    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-kube-api-access-blm2l" pod "fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" (UID: "fe83fc60-2039-4d54-a1f3-8f3e5351d9d5"). InnerVolumeSpecName "kube-api-access-blm2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:40:09.204566 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.204442    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:40:09.204566 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.204479    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blm2l\" (UniqueName: \"kubernetes.io/projected/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-kube-api-access-blm2l\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:40:09.204566 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.204495    4931 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5-webhook-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:40:09.477815 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.477567    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-dssdt" event={"ID":"8d282f7e-e082-4cc0-ac9b-39880957b56e","Type":"ContainerStarted","Data":"72ee92e1a84831a5c2296e31dc5222f7623dac9f1c60a39a9179fcf5a4215da0"}
Nov 01 08:40:09.477815 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.477746    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:09.480547 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.480515    4931 generic.go:356] "Generic (PLEG): container finished" podID="fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" containerID="e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2" exitCode=0
Nov 01 08:40:09.480687 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.480645    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fb4fcc885-r994f" event={"ID":"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5","Type":"ContainerDied","Data":"e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2"}
Nov 01 08:40:09.480742 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.480693    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fb4fcc885-r994f" event={"ID":"fe83fc60-2039-4d54-a1f3-8f3e5351d9d5","Type":"ContainerDied","Data":"cef0a567a0b4b4fdb70b5f7208782c08b140a8f41014adeb3f22189fa2638385"}
Nov 01 08:40:09.480742 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.480723    4931 scope.go:122] "RemoveContainer" containerID="e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2"
Nov 01 08:40:09.480882 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.480858    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fb4fcc885-r994f"
Nov 01 08:40:09.496836 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.496811    4931 scope.go:122] "RemoveContainer" containerID="e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2"
Nov 01 08:40:09.497619 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:40:09.497591    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2\": container with ID starting with e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2 not found: ID does not exist" containerID="e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2"
Nov 01 08:40:09.497724 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.497637    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2"} err="failed to get container status \"e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2\": rpc error: code = NotFound desc = could not find container \"e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2\": container with ID starting with e409dae02b38006975ebc173b4839d9ab474bd7a0ab9cb2de5c78db2cdac1bf2 not found: ID does not exist"
Nov 01 08:40:09.592871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.592113    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-dssdt" podStartSLOduration=2.299672417 podStartE2EDuration="6.592098232s" totalImagesPullingTime="4.292425815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:09.590580236 +0000 UTC m=+479.705453276" watchObservedRunningTime="2026-11-01 08:40:09.592098232 +0000 UTC m=+479.706971200"
Nov 01 08:40:09.699802 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.699731    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fb4fcc885-r994f"]
Nov 01 08:40:09.705349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:09.705315    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fb4fcc885-r994f"]
Nov 01 08:40:10.051983 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:10.051944    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" path="/var/lib/kubelet/pods/fe83fc60-2039-4d54-a1f3-8f3e5351d9d5/volumes"
Nov 01 08:40:10.489376 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:10.489335    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd" event={"ID":"e013612f-52d5-4e9a-9e1e-5090e63a6fde","Type":"ContainerStarted","Data":"f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4"}
Nov 01 08:40:10.489893 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:10.489438    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:10.491523 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:10.491490    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-z247t" event={"ID":"cc0775c9-babb-41b8-8db2-f958a8f69c95","Type":"ContainerStarted","Data":"38ddb2f4947fb9677cf48955a9b95d87c71525c68b100b64493146a7dc6ed089"}
Nov 01 08:40:10.491651 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:10.491631    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:10.604568 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:10.602711    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd" podStartSLOduration=1.583216918 podStartE2EDuration="7.602688597s" totalImagesPullingTime="6.019471679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:10.601768278 +0000 UTC m=+480.716641273" watchObservedRunningTime="2026-11-01 08:40:10.602688597 +0000 UTC m=+480.717561572"
Nov 01 08:40:10.716939 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:10.716877    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-z247t" podStartSLOduration=2.013231265 podStartE2EDuration="7.71684928s" totalImagesPullingTime="5.703618015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:10.713371734 +0000 UTC m=+480.828244752" watchObservedRunningTime="2026-11-01 08:40:10.71684928 +0000 UTC m=+480.831722249"
Nov 01 08:40:17.540889 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:17.540846    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b" event={"ID":"f5f45383-80fa-4ee1-9556-96ba0b54a988","Type":"ContainerStarted","Data":"3c11f90e275a04684f11982850d714b30dc10d41ede210f31737d9ebbae2013f"}
Nov 01 08:40:17.541307 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:17.540988    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:17.654520 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:17.654464    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b" podStartSLOduration=2.022334996 podStartE2EDuration="16.654450352s" totalImagesPullingTime="14.632115356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:17.6528367 +0000 UTC m=+487.767709667" watchObservedRunningTime="2026-11-01 08:40:17.654450352 +0000 UTC m=+487.769323320"
Nov 01 08:40:20.495941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:20.495864    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-dssdt"
Nov 01 08:40:28.551794 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:28.551752    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-dcm8b"
Nov 01 08:40:31.024054 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:40:31.023987    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:40:41.502178 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:41.502136    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:40:41.505241 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:41.505218    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-z247t"
Nov 01 08:40:45.835305 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.809351    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-x8wfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": read tcp 10.133.0.2:33032->10.133.0.32:15021: read: connection reset by peer" start-of-body=
Nov 01 08:40:45.835305 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.809434    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": read tcp 10.133.0.2:33032->10.133.0.32:15021: read: connection reset by peer"
Nov 01 08:40:45.848529 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.848503    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-z4g8r"]
Nov 01 08:40:45.848935 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.848922    4931 cpu_manager.go:414] "containerMap: removing container" podUID="fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" containerName="manager"
Nov 01 08:40:45.848983 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.848936    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" containerName="manager"
Nov 01 08:40:45.849017 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.849004    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe83fc60-2039-4d54-a1f3-8f3e5351d9d5" containerName="manager"
Nov 01 08:40:45.881855 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.881826    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r"
Nov 01 08:40:45.905701 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.905651    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-z4g8r"]
Nov 01 08:40:45.950181 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.950141    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wlrzh\""
Nov 01 08:40:45.955557 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:45.955531    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ph2\" (UniqueName: \"kubernetes.io/projected/b0b07e9b-6ca4-4520-866c-72a22fd66fab-kube-api-access-56ph2\") pod \"authorino-6f6944bcdd-z4g8r\" (UID: \"b0b07e9b-6ca4-4520-866c-72a22fd66fab\") " pod="kuadrant-system/authorino-6f6944bcdd-z4g8r"
Nov 01 08:40:46.036118 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.035997    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"]
Nov 01 08:40:46.056591 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.056559    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ph2\" (UniqueName: \"kubernetes.io/projected/b0b07e9b-6ca4-4520-866c-72a22fd66fab-kube-api-access-56ph2\") pod \"authorino-6f6944bcdd-z4g8r\" (UID: \"b0b07e9b-6ca4-4520-866c-72a22fd66fab\") " pod="kuadrant-system/authorino-6f6944bcdd-z4g8r"
Nov 01 08:40:46.057247 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.057220    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.095835 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.095803    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"]
Nov 01 08:40:46.107422 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.107393    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 08:40:46.107634 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.107619    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 08:40:46.107690 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.107671    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7zt8\""
Nov 01 08:40:46.143601 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.143568    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ph2\" (UniqueName: \"kubernetes.io/projected/b0b07e9b-6ca4-4520-866c-72a22fd66fab-kube-api-access-56ph2\") pod \"authorino-6f6944bcdd-z4g8r\" (UID: \"b0b07e9b-6ca4-4520-866c-72a22fd66fab\") " pod="kuadrant-system/authorino-6f6944bcdd-z4g8r"
Nov 01 08:40:46.157541 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.157514    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-metrics-certs\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.157694 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.157585    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-maas-api-tls\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.157745 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.157686    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgxx\" (UniqueName: \"kubernetes.io/projected/034aef14-4a57-4833-bfb8-de72c3882304-kube-api-access-gsgxx\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.157745 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.157716    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034aef14-4a57-4833-bfb8-de72c3882304-openshift-service-ca\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.207326 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.207290    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-z4g8r"]
Nov 01 08:40:46.207534 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.207522    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r"
Nov 01 08:40:46.258829 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.258793    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-metrics-certs\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.259002 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.258874    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-maas-api-tls\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.259002 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.258920    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgxx\" (UniqueName: \"kubernetes.io/projected/034aef14-4a57-4833-bfb8-de72c3882304-kube-api-access-gsgxx\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.259002 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.258946    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034aef14-4a57-4833-bfb8-de72c3882304-openshift-service-ca\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.259659 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.259598    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034aef14-4a57-4833-bfb8-de72c3882304-openshift-service-ca\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.261540 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.261511    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-maas-api-tls\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.261633 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.261576    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-metrics-certs\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.318011 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.317944    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgxx\" (UniqueName: \"kubernetes.io/projected/034aef14-4a57-4833-bfb8-de72c3882304-kube-api-access-gsgxx\") pod \"maas-api-7dcf475b5f-dm9cv\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.401074 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.401017    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:46.599623 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.599599    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-z4g8r"]
Nov 01 08:40:46.601204 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:40:46.601175    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0b07e9b_6ca4_4520_866c_72a22fd66fab.slice/crio-b0d39a1f2fc91e90f6f395e4bb148a5a7ab54fac29688c9c9eaa1153f969491b WatchSource:0}: Error finding container b0d39a1f2fc91e90f6f395e4bb148a5a7ab54fac29688c9c9eaa1153f969491b: Status 404 returned error can't find the container with id b0d39a1f2fc91e90f6f395e4bb148a5a7ab54fac29688c9c9eaa1153f969491b
Nov 01 08:40:46.712176 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.712139    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r" event={"ID":"b0b07e9b-6ca4-4520-866c-72a22fd66fab","Type":"ContainerStarted","Data":"b0d39a1f2fc91e90f6f395e4bb148a5a7ab54fac29688c9c9eaa1153f969491b"}
Nov 01 08:40:46.713855 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.713819    4931 generic.go:356] "Generic (PLEG): container finished" podID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerID="78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0" exitCode=0
Nov 01 08:40:46.714001 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.713913    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" event={"ID":"91aedd00-ab73-46a1-8ef7-11d6440a4706","Type":"ContainerDied","Data":"78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0"}
Nov 01 08:40:46.714342 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.714328    4931 scope.go:122] "RemoveContainer" containerID="78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0"
Nov 01 08:40:46.715476 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.715444    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:40:46.715596 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.715519    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:40:46.715596 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.715550    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:40:46.729446 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.729422    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d8c64554d-xpwdr"]
Nov 01 08:40:46.761505 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.761358    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:40:46.788476 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.788447    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d8c64554d-xpwdr"]
Nov 01 08:40:46.795910 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.795782    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"]
Nov 01 08:40:46.798752 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:40:46.798725    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034aef14_4a57_4833_bfb8_de72c3882304.slice/crio-20a827522baa064fb4b88f22408e600a0fff466b2ea7166f60ec2517b6ada6eb WatchSource:0}: Error finding container 20a827522baa064fb4b88f22408e600a0fff466b2ea7166f60ec2517b6ada6eb: Status 404 returned error can't find the container with id 20a827522baa064fb4b88f22408e600a0fff466b2ea7166f60ec2517b6ada6eb
Nov 01 08:40:46.811696 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.811674    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 08:40:46.865189 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.865153    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bt9d\" (UniqueName: \"kubernetes.io/projected/1e442467-1537-43ae-a876-842360b3f522-kube-api-access-7bt9d\") pod \"authorino-6d8c64554d-xpwdr\" (UID: \"1e442467-1537-43ae-a876-842360b3f522\") " pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:40:46.865805 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.865300    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e442467-1537-43ae-a876-842360b3f522-tls-cert\") pod \"authorino-6d8c64554d-xpwdr\" (UID: \"1e442467-1537-43ae-a876-842360b3f522\") " pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:40:46.966570 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.966528    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e442467-1537-43ae-a876-842360b3f522-tls-cert\") pod \"authorino-6d8c64554d-xpwdr\" (UID: \"1e442467-1537-43ae-a876-842360b3f522\") " pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:40:46.966754 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.966584    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bt9d\" (UniqueName: \"kubernetes.io/projected/1e442467-1537-43ae-a876-842360b3f522-kube-api-access-7bt9d\") pod \"authorino-6d8c64554d-xpwdr\" (UID: \"1e442467-1537-43ae-a876-842360b3f522\") " pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:40:46.969268 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:46.969244    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e442467-1537-43ae-a876-842360b3f522-tls-cert\") pod \"authorino-6d8c64554d-xpwdr\" (UID: \"1e442467-1537-43ae-a876-842360b3f522\") " pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:40:47.025730 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:47.025682    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bt9d\" (UniqueName: \"kubernetes.io/projected/1e442467-1537-43ae-a876-842360b3f522-kube-api-access-7bt9d\") pod \"authorino-6d8c64554d-xpwdr\" (UID: \"1e442467-1537-43ae-a876-842360b3f522\") " pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:40:47.088951 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:47.088919    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:40:47.493354 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:47.493322    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d8c64554d-xpwdr"]
Nov 01 08:40:47.494498 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:40:47.494470    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e442467_1537_43ae_a876_842360b3f522.slice/crio-62cd81d2da921d12d0acba7287451de655dec772962ea41fd7657404637a6559 WatchSource:0}: Error finding container 62cd81d2da921d12d0acba7287451de655dec772962ea41fd7657404637a6559: Status 404 returned error can't find the container with id 62cd81d2da921d12d0acba7287451de655dec772962ea41fd7657404637a6559
Nov 01 08:40:47.720691 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:47.720656    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d8c64554d-xpwdr" event={"ID":"1e442467-1537-43ae-a876-842360b3f522","Type":"ContainerStarted","Data":"62cd81d2da921d12d0acba7287451de655dec772962ea41fd7657404637a6559"}
Nov 01 08:40:47.721967 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:47.721927    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv" event={"ID":"034aef14-4a57-4833-bfb8-de72c3882304","Type":"ContainerStarted","Data":"20a827522baa064fb4b88f22408e600a0fff466b2ea7166f60ec2517b6ada6eb"}
Nov 01 08:40:47.723976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:47.723948    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" event={"ID":"91aedd00-ab73-46a1-8ef7-11d6440a4706","Type":"ContainerStarted","Data":"31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6"}
Nov 01 08:40:48.475573 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:48.475486    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:40:48.477486 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:48.477457    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-x8wfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:40:48.477719 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:48.477690    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 01 08:40:49.470167 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:49.469473    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-x8wfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:40:49.470167 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:49.469537    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 01 08:40:49.740070 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:49.739958    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d8c64554d-xpwdr" event={"ID":"1e442467-1537-43ae-a876-842360b3f522","Type":"ContainerStarted","Data":"7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc"}
Nov 01 08:40:49.741757 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:49.741732    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r" event={"ID":"b0b07e9b-6ca4-4520-866c-72a22fd66fab","Type":"ContainerStarted","Data":"9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8"}
Nov 01 08:40:49.741901 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:49.741837    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r" podUID="b0b07e9b-6ca4-4520-866c-72a22fd66fab" containerName="authorino" containerID="cri-o://9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8" gracePeriod=30
Nov 01 08:40:49.850729 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:49.850676    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d8c64554d-xpwdr" podStartSLOduration=1.793957821 podStartE2EDuration="3.850661093s" totalImagesPullingTime="2.056703272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:49.849799681 +0000 UTC m=+519.964672671" watchObservedRunningTime="2026-11-01 08:40:49.850661093 +0000 UTC m=+519.965534061"
Nov 01 08:40:49.986061 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:49.974356    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r" podStartSLOduration=2.791259442 podStartE2EDuration="4.974335645s" totalImagesPullingTime="2.183076203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:49.97229294 +0000 UTC m=+520.087165918" watchObservedRunningTime="2026-11-01 08:40:49.974335645 +0000 UTC m=+520.089208615"
Nov 01 08:40:50.127391 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.127358    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r"
Nov 01 08:40:50.212784 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.212752    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b07e9b-6ca4-4520-866c-72a22fd66fab-kube-api-access-56ph2\" (UniqueName: \"kubernetes.io/projected/b0b07e9b-6ca4-4520-866c-72a22fd66fab-kube-api-access-56ph2\") pod \"b0b07e9b-6ca4-4520-866c-72a22fd66fab\" (UID: \"b0b07e9b-6ca4-4520-866c-72a22fd66fab\") "
Nov 01 08:40:50.215580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.215549    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b07e9b-6ca4-4520-866c-72a22fd66fab-kube-api-access-56ph2" pod "b0b07e9b-6ca4-4520-866c-72a22fd66fab" (UID: "b0b07e9b-6ca4-4520-866c-72a22fd66fab"). InnerVolumeSpecName "kube-api-access-56ph2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:40:50.313851 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.313806    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ph2\" (UniqueName: \"kubernetes.io/projected/b0b07e9b-6ca4-4520-866c-72a22fd66fab-kube-api-access-56ph2\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:40:50.466388 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.463422    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-x8wfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:40:50.466388 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.463505    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 01 08:40:50.750436 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.750330    4931 generic.go:356] "Generic (PLEG): container finished" podID="b0b07e9b-6ca4-4520-866c-72a22fd66fab" containerID="9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8" exitCode=0
Nov 01 08:40:50.750436 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.750377    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r"
Nov 01 08:40:50.750436 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.750419    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r" event={"ID":"b0b07e9b-6ca4-4520-866c-72a22fd66fab","Type":"ContainerDied","Data":"9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8"}
Nov 01 08:40:50.751729 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.750459    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-z4g8r" event={"ID":"b0b07e9b-6ca4-4520-866c-72a22fd66fab","Type":"ContainerDied","Data":"b0d39a1f2fc91e90f6f395e4bb148a5a7ab54fac29688c9c9eaa1153f969491b"}
Nov 01 08:40:50.751729 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.750482    4931 scope.go:122] "RemoveContainer" containerID="9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8"
Nov 01 08:40:50.790306 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.790203    4931 scope.go:122] "RemoveContainer" containerID="9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8"
Nov 01 08:40:50.790641 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:40:50.790626    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8\": container with ID starting with 9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8 not found: ID does not exist" containerID="9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8"
Nov 01 08:40:50.790745 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.790723    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8"} err="failed to get container status \"9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8\": rpc error: code = NotFound desc = could not find container \"9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8\": container with ID starting with 9292a3feb9c05ddc961b680390ba4639dbbb3ad86b2b91994f40f526aaeaaed8 not found: ID does not exist"
Nov 01 08:40:50.907563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.907507    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-z4g8r"]
Nov 01 08:40:50.914464 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:50.914426    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-z4g8r"]
Nov 01 08:40:51.485047 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:51.484962    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-x8wfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:40:51.485222 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:51.485044    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 01 08:40:51.757156 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:51.757054    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv" event={"ID":"034aef14-4a57-4833-bfb8-de72c3882304","Type":"ContainerStarted","Data":"1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb"}
Nov 01 08:40:51.757573 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:51.757209    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:40:51.881051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:51.880597    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv" podStartSLOduration=2.242307812 podStartE2EDuration="6.880577674s" totalImagesPullingTime="4.638269862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:51.877509306 +0000 UTC m=+521.992382288" watchObservedRunningTime="2026-11-01 08:40:51.880577674 +0000 UTC m=+521.995450646"
Nov 01 08:40:52.046952 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:52.046873    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b07e9b-6ca4-4520-866c-72a22fd66fab" path="/var/lib/kubelet/pods/b0b07e9b-6ca4-4520-866c-72a22fd66fab/volumes"
Nov 01 08:40:52.480055 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:52.463465    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-x8wfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 01 08:40:52.480055 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:52.463528    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 01 08:40:53.466689 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:53.466656    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:40:53.467103 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:53.466971    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:40:53.467679 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:53.467659    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 08:40:57.770166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:40:57.770139    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 08:41:01.022548 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:41:01.022509    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:41:10.971619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:10.971583    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"]
Nov 01 08:41:10.972212 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:10.972192    4931 cpu_manager.go:414] "containerMap: removing container" podUID="b0b07e9b-6ca4-4520-866c-72a22fd66fab" containerName="authorino"
Nov 01 08:41:10.972309 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:10.972214    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b07e9b-6ca4-4520-866c-72a22fd66fab" containerName="authorino"
Nov 01 08:41:10.972372 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:10.972352    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b07e9b-6ca4-4520-866c-72a22fd66fab" containerName="authorino"
Nov 01 08:41:10.976938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:10.976915    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:11.031933 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.031890    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"]
Nov 01 08:41:11.103677 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.103639    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c07de17d-572b-4a2c-a44c-b74048d7ec0a-cert\") pod \"llmisvc-controller-manager-6466c554d8-m97nw\" (UID: \"c07de17d-572b-4a2c-a44c-b74048d7ec0a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:11.103855 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.103703    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798cv\" (UniqueName: \"kubernetes.io/projected/c07de17d-572b-4a2c-a44c-b74048d7ec0a-kube-api-access-798cv\") pod \"llmisvc-controller-manager-6466c554d8-m97nw\" (UID: \"c07de17d-572b-4a2c-a44c-b74048d7ec0a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:11.204776 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.204740    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c07de17d-572b-4a2c-a44c-b74048d7ec0a-cert\") pod \"llmisvc-controller-manager-6466c554d8-m97nw\" (UID: \"c07de17d-572b-4a2c-a44c-b74048d7ec0a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:11.204939 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.204818    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798cv\" (UniqueName: \"kubernetes.io/projected/c07de17d-572b-4a2c-a44c-b74048d7ec0a-kube-api-access-798cv\") pod \"llmisvc-controller-manager-6466c554d8-m97nw\" (UID: \"c07de17d-572b-4a2c-a44c-b74048d7ec0a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:11.207344 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.207323    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c07de17d-572b-4a2c-a44c-b74048d7ec0a-cert\") pod \"llmisvc-controller-manager-6466c554d8-m97nw\" (UID: \"c07de17d-572b-4a2c-a44c-b74048d7ec0a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:11.263843 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.263764    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-798cv\" (UniqueName: \"kubernetes.io/projected/c07de17d-572b-4a2c-a44c-b74048d7ec0a-kube-api-access-798cv\") pod \"llmisvc-controller-manager-6466c554d8-m97nw\" (UID: \"c07de17d-572b-4a2c-a44c-b74048d7ec0a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:11.297668 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.297631    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:11.662398 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.662366    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"]
Nov 01 08:41:11.663918 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:41:11.663869    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07de17d_572b_4a2c_a44c_b74048d7ec0a.slice/crio-c82155b94e8efc2efcb5c852dfaa3b1d6d2b2e53797ea70f3beac80d1711848a WatchSource:0}: Error finding container c82155b94e8efc2efcb5c852dfaa3b1d6d2b2e53797ea70f3beac80d1711848a: Status 404 returned error can't find the container with id c82155b94e8efc2efcb5c852dfaa3b1d6d2b2e53797ea70f3beac80d1711848a
Nov 01 08:41:11.865500 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:11.865460    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw" event={"ID":"c07de17d-572b-4a2c-a44c-b74048d7ec0a","Type":"ContainerStarted","Data":"c82155b94e8efc2efcb5c852dfaa3b1d6d2b2e53797ea70f3beac80d1711848a"}
Nov 01 08:41:12.873073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:12.873006    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw" event={"ID":"c07de17d-572b-4a2c-a44c-b74048d7ec0a","Type":"ContainerStarted","Data":"07d347ea0b777845a2548c18b025f5196dbf4743bc6be78d61d12f2b6a58e1f2"}
Nov 01 08:41:12.873562 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:12.873158    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:12.985649 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:12.985584    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw" podStartSLOduration=2.166815721 podStartE2EDuration="2.985569963s" totalImagesPullingTime="818.754242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:41:12.984474962 +0000 UTC m=+543.099347930" watchObservedRunningTime="2026-11-01 08:41:12.985569963 +0000 UTC m=+543.100442931"
Nov 01 08:41:31.021694 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:41:31.021645    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:41:43.880426 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:43.880395    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m97nw"
Nov 01 08:41:44.046760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.046732    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"]
Nov 01 08:41:44.046969 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.046933    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd" podUID="e013612f-52d5-4e9a-9e1e-5090e63a6fde" containerName="manager" containerID="cri-o://f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4" gracePeriod=30
Nov 01 08:41:44.298595 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.298571    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:41:44.383530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.383497    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e013612f-52d5-4e9a-9e1e-5090e63a6fde-cert\" (UniqueName: \"kubernetes.io/secret/e013612f-52d5-4e9a-9e1e-5090e63a6fde-cert\") pod \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\" (UID: \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\") "
Nov 01 08:41:44.383530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.383535    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e013612f-52d5-4e9a-9e1e-5090e63a6fde-kube-api-access-d92wv\" (UniqueName: \"kubernetes.io/projected/e013612f-52d5-4e9a-9e1e-5090e63a6fde-kube-api-access-d92wv\") pod \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\" (UID: \"e013612f-52d5-4e9a-9e1e-5090e63a6fde\") "
Nov 01 08:41:44.385776 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.385750    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e013612f-52d5-4e9a-9e1e-5090e63a6fde-cert" pod "e013612f-52d5-4e9a-9e1e-5090e63a6fde" (UID: "e013612f-52d5-4e9a-9e1e-5090e63a6fde"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:41:44.385926 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.385786    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e013612f-52d5-4e9a-9e1e-5090e63a6fde-kube-api-access-d92wv" pod "e013612f-52d5-4e9a-9e1e-5090e63a6fde" (UID: "e013612f-52d5-4e9a-9e1e-5090e63a6fde"). InnerVolumeSpecName "kube-api-access-d92wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:41:44.484442 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.484364    4931 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e013612f-52d5-4e9a-9e1e-5090e63a6fde-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:41:44.484442 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:44.484389    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d92wv\" (UniqueName: \"kubernetes.io/projected/e013612f-52d5-4e9a-9e1e-5090e63a6fde-kube-api-access-d92wv\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:41:45.043848 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.043813    4931 generic.go:356] "Generic (PLEG): container finished" podID="e013612f-52d5-4e9a-9e1e-5090e63a6fde" containerID="f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4" exitCode=0
Nov 01 08:41:45.044313 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.043858    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"
Nov 01 08:41:45.044313 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.043870    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd" event={"ID":"e013612f-52d5-4e9a-9e1e-5090e63a6fde","Type":"ContainerDied","Data":"f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4"}
Nov 01 08:41:45.044313 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.043923    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd" event={"ID":"e013612f-52d5-4e9a-9e1e-5090e63a6fde","Type":"ContainerDied","Data":"5a977cfeb4682b92b722f34b47a5cb9c43d81ea0664c3c826d4c0c88e6c62d0f"}
Nov 01 08:41:45.044313 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.043953    4931 scope.go:122] "RemoveContainer" containerID="f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4"
Nov 01 08:41:45.053653 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.053496    4931 scope.go:122] "RemoveContainer" containerID="f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4"
Nov 01 08:41:45.053781 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:41:45.053761    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4\": container with ID starting with f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4 not found: ID does not exist" containerID="f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4"
Nov 01 08:41:45.053841 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.053794    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4"} err="failed to get container status \"f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4\": rpc error: code = NotFound desc = could not find container \"f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4\": container with ID starting with f025fc5c4807721cc2d9860b86a4944139461ba646fbd1794832f5c7e87a9ed4 not found: ID does not exist"
Nov 01 08:41:45.203420 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.203387    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"]
Nov 01 08:41:45.211576 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:45.211550    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gdpd"]
Nov 01 08:41:46.047262 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:41:46.047222    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e013612f-52d5-4e9a-9e1e-5090e63a6fde" path="/var/lib/kubelet/pods/e013612f-52d5-4e9a-9e1e-5090e63a6fde/volumes"
Nov 01 08:42:01.023211 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:42:01.023128    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:42:10.086934 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:42:10.086902    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:42:10.088759 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:42:10.088735    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:42:10.090749 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:42:10.090728    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:42:10.092733 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:42:10.092717    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:42:31.022387 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:42:31.022347    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:43:01.022363 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:43:01.022318    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:43:05.905364 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:05.905323    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"]
Nov 01 08:43:05.905863 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:05.905757    4931 cpu_manager.go:414] "containerMap: removing container" podUID="e013612f-52d5-4e9a-9e1e-5090e63a6fde" containerName="manager"
Nov 01 08:43:05.905863 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:05.905770    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e013612f-52d5-4e9a-9e1e-5090e63a6fde" containerName="manager"
Nov 01 08:43:05.905977 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:05.905878    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e013612f-52d5-4e9a-9e1e-5090e63a6fde" containerName="manager"
Nov 01 08:43:05.909427 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:05.909405    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047325 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047291    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047503 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047335    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047503 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047410    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047503 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047437    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047503 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047456    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047721 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047517    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkq4h\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-kube-api-access-qkq4h\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047721 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047585    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047721 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047615    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.047721 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.047645    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.109663 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.109629    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"]
Nov 01 08:43:06.148846 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.148809    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.148846 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.148849    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149150 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.148871    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149150 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.148907    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkq4h\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-kube-api-access-qkq4h\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149150 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.148936    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149150 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.149092    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149367 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.149154    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149367 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.149203    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149367 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.149238    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149669 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.149650    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149748 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.149685    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149748 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.149724    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.149835 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.149744    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.150159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.150141    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.151590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.151566    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.152122 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.152104    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.207683 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.207614    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.207892 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.207870    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkq4h\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-kube-api-access-qkq4h\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fvjhx\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.222559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.222532    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:06.647860 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.647833    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"]
Nov 01 08:43:06.648504 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:43:06.648478    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7ae9a2_9efa_4c86_91f3_3c84f1d1bdce.slice/crio-461d5afc6b163000dc2ba66a8a34d6db21b438bda0198c2cc2cb90caf6490e60 WatchSource:0}: Error finding container 461d5afc6b163000dc2ba66a8a34d6db21b438bda0198c2cc2cb90caf6490e60: Status 404 returned error can't find the container with id 461d5afc6b163000dc2ba66a8a34d6db21b438bda0198c2cc2cb90caf6490e60
Nov 01 08:43:06.650652 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.650618    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:43:06.650726 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.650679    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:43:06.650726 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:06.650708    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:43:07.496396 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:07.496359    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" event={"ID":"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce","Type":"ContainerStarted","Data":"aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded"}
Nov 01 08:43:07.496814 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:07.496419    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" event={"ID":"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce","Type":"ContainerStarted","Data":"461d5afc6b163000dc2ba66a8a34d6db21b438bda0198c2cc2cb90caf6490e60"}
Nov 01 08:43:08.222719 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:08.222684    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:08.224332 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:08.224306    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 08:43:08.224465 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:08.224353    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 08:43:09.223520 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:09.223480    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 08:43:09.224057 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:09.223556    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 08:43:10.222937 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:10.222897    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 08:43:10.223147 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:10.222964    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 08:43:11.223851 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:11.223805    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 08:43:11.224319 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:11.223862    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 08:43:12.223221 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:12.223181    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 08:43:12.223479 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:12.223250    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 08:43:13.223875 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:13.223835    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 08:43:13.224398 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:13.223961    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 08:43:14.222902 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:14.222866    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 08:43:14.223133 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:14.222948    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 08:43:15.826118 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:15.826077    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:15.826605 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:15.826461    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:15.827330 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:15.827291    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 08:43:15.991895 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:15.991840    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podStartSLOduration=10.991825652 podStartE2EDuration="10.991825652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:07.677289978 +0000 UTC m=+657.792162946" watchObservedRunningTime="2026-11-01 08:43:15.991825652 +0000 UTC m=+666.106698619"
Nov 01 08:43:22.000405 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.000305    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d8c64554d-xpwdr"]
Nov 01 08:43:22.000900 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.000493    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6d8c64554d-xpwdr" podUID="1e442467-1537-43ae-a876-842360b3f522" containerName="authorino" containerID="cri-o://7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc" gracePeriod=30
Nov 01 08:43:22.264370 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.264308    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:43:22.320499 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.320473    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e442467-1537-43ae-a876-842360b3f522-tls-cert\" (UniqueName: \"kubernetes.io/secret/1e442467-1537-43ae-a876-842360b3f522-tls-cert\") pod \"1e442467-1537-43ae-a876-842360b3f522\" (UID: \"1e442467-1537-43ae-a876-842360b3f522\") "
Nov 01 08:43:22.320499 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.320507    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e442467-1537-43ae-a876-842360b3f522-kube-api-access-7bt9d\" (UniqueName: \"kubernetes.io/projected/1e442467-1537-43ae-a876-842360b3f522-kube-api-access-7bt9d\") pod \"1e442467-1537-43ae-a876-842360b3f522\" (UID: \"1e442467-1537-43ae-a876-842360b3f522\") "
Nov 01 08:43:22.322694 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.322669    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e442467-1537-43ae-a876-842360b3f522-kube-api-access-7bt9d" pod "1e442467-1537-43ae-a876-842360b3f522" (UID: "1e442467-1537-43ae-a876-842360b3f522"). InnerVolumeSpecName "kube-api-access-7bt9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:43:22.333490 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.333464    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e442467-1537-43ae-a876-842360b3f522-tls-cert" pod "1e442467-1537-43ae-a876-842360b3f522" (UID: "1e442467-1537-43ae-a876-842360b3f522"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:43:22.422106 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.422070    4931 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e442467-1537-43ae-a876-842360b3f522-tls-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:43:22.422106 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.422099    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bt9d\" (UniqueName: \"kubernetes.io/projected/1e442467-1537-43ae-a876-842360b3f522-kube-api-access-7bt9d\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:43:22.589070 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.588964    4931 generic.go:356] "Generic (PLEG): container finished" podID="1e442467-1537-43ae-a876-842360b3f522" containerID="7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc" exitCode=0
Nov 01 08:43:22.589070 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.589016    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8c64554d-xpwdr"
Nov 01 08:43:22.589070 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.589048    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d8c64554d-xpwdr" event={"ID":"1e442467-1537-43ae-a876-842360b3f522","Type":"ContainerDied","Data":"7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc"}
Nov 01 08:43:22.589351 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.589105    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d8c64554d-xpwdr" event={"ID":"1e442467-1537-43ae-a876-842360b3f522","Type":"ContainerDied","Data":"62cd81d2da921d12d0acba7287451de655dec772962ea41fd7657404637a6559"}
Nov 01 08:43:22.589351 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.589124    4931 scope.go:122] "RemoveContainer" containerID="7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc"
Nov 01 08:43:22.600092 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.600072    4931 scope.go:122] "RemoveContainer" containerID="7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc"
Nov 01 08:43:22.600389 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:43:22.600372    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc\": container with ID starting with 7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc not found: ID does not exist" containerID="7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc"
Nov 01 08:43:22.600457 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.600406    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc"} err="failed to get container status \"7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc\": rpc error: code = NotFound desc = could not find container \"7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc\": container with ID starting with 7bdf131d55468a2d4d0a5ba84e5caa9b584cb4fc958667662bb804cadab7c3fc not found: ID does not exist"
Nov 01 08:43:22.746456 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.746423    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d8c64554d-xpwdr"]
Nov 01 08:43:22.794108 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:22.794074    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d8c64554d-xpwdr"]
Nov 01 08:43:24.046079 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:43:24.046020    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e442467-1537-43ae-a876-842360b3f522" path="/var/lib/kubelet/pods/1e442467-1537-43ae-a876-842360b3f522/volumes"
Nov 01 08:43:31.022313 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:43:31.022276    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:44:01.025590 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:44:01.025551    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:44:03.807055 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.807005    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"]
Nov 01 08:44:03.807484 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.807443    4931 cpu_manager.go:414] "containerMap: removing container" podUID="1e442467-1537-43ae-a876-842360b3f522" containerName="authorino"
Nov 01 08:44:03.807484 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.807455    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e442467-1537-43ae-a876-842360b3f522" containerName="authorino"
Nov 01 08:44:03.807709 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.807686    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e442467-1537-43ae-a876-842360b3f522" containerName="authorino"
Nov 01 08:44:03.811386 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.811367    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:03.865777 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.865744    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"]
Nov 01 08:44:03.889612 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.889581    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd\" (UID: \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\") " pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:03.889788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.889700    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp294\" (UniqueName: \"kubernetes.io/projected/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-kube-api-access-fp294\") pod \"payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd\" (UID: \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\") " pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:03.909951 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.909922    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f814ac10\""
Nov 01 08:44:03.909951 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.909923    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f814ac10-dockercfg-trsw7\""
Nov 01 08:44:03.990219 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.990186    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp294\" (UniqueName: \"kubernetes.io/projected/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-kube-api-access-fp294\") pod \"payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd\" (UID: \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\") " pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:03.990397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.990233    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd\" (UID: \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\") " pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:03.990945 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:03.990927    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd\" (UID: \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\") " pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:04.056192 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.056159    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp294\" (UniqueName: \"kubernetes.io/projected/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-kube-api-access-fp294\") pod \"payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd\" (UID: \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\") " pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:04.126277 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.126248    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:04.505861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.505826    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"]
Nov 01 08:44:04.506817 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:44:04.506795    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcf7dc5_d8a6_4178_9d15_a18442dd6d53.slice/crio-303cfb26203f41d2d3f9610ff7eb9e9c2868be77f1b9f9be37341e518255d16a WatchSource:0}: Error finding container 303cfb26203f41d2d3f9610ff7eb9e9c2868be77f1b9f9be37341e518255d16a: Status 404 returned error can't find the container with id 303cfb26203f41d2d3f9610ff7eb9e9c2868be77f1b9f9be37341e518255d16a
Nov 01 08:44:04.508709 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.508694    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:44:04.817138 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.817057    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd" event={"ID":"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53","Type":"ContainerStarted","Data":"303cfb26203f41d2d3f9610ff7eb9e9c2868be77f1b9f9be37341e518255d16a"}
Nov 01 08:44:04.917186 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.917155    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"]
Nov 01 08:44:04.922013 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.921995    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:04.973226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.973187    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5c25ae7a-serving-cert\""
Nov 01 08:44:04.975960 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.975932    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"]
Nov 01 08:44:04.997969 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.997943    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:04.998145 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.997985    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtrn\" (UniqueName: \"kubernetes.io/projected/1abaa356-6613-4f4d-8f41-d5bc02411535-kube-api-access-pgtrn\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:04.998145 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.998009    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abaa356-6613-4f4d-8f41-d5bc02411535-openshift-service-ca\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:04.998279 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:04.998183    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-metrics-certs\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.098783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.098682    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abaa356-6613-4f4d-8f41-d5bc02411535-openshift-service-ca\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.098783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.098764    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-metrics-certs\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.099042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.098918    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.099042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.098982    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtrn\" (UniqueName: \"kubernetes.io/projected/1abaa356-6613-4f4d-8f41-d5bc02411535-kube-api-access-pgtrn\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.099259 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:44:05.099240    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-serving-cert: secret "maas-api-e2e-ait-5c25ae7a-serving-cert" not found
Nov 01 08:44:05.099360 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:44:05.099309    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls podName:1abaa356-6613-4f4d-8f41-d5bc02411535 nodeName:}" failed. No retries permitted until 2026-11-01 08:44:05.599289779 +0000 UTC m=+715.714162725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls") pod "maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b" (UID: "1abaa356-6613-4f4d-8f41-d5bc02411535") : secret "maas-api-e2e-ait-5c25ae7a-serving-cert" not found
Nov 01 08:44:05.099510 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.099480    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abaa356-6613-4f4d-8f41-d5bc02411535-openshift-service-ca\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.101605 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.101584    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-metrics-certs\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.158803 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.158774    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtrn\" (UniqueName: \"kubernetes.io/projected/1abaa356-6613-4f4d-8f41-d5bc02411535-kube-api-access-pgtrn\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.603478 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.603439    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.606057 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.606001    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls\") pod \"maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:05.835380 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:05.835343    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:06.228999 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:06.228957    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"]
Nov 01 08:44:06.232673 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:44:06.232643    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abaa356_6613_4f4d_8f41_d5bc02411535.slice/crio-20fdf7ba8da6c22980f889d805a5300cbc5f69a8ef9f8b4ae13cf29f62e9d3ba WatchSource:0}: Error finding container 20fdf7ba8da6c22980f889d805a5300cbc5f69a8ef9f8b4ae13cf29f62e9d3ba: Status 404 returned error can't find the container with id 20fdf7ba8da6c22980f889d805a5300cbc5f69a8ef9f8b4ae13cf29f62e9d3ba
Nov 01 08:44:06.833051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:06.833007    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b" event={"ID":"1abaa356-6613-4f4d-8f41-d5bc02411535","Type":"ContainerStarted","Data":"20fdf7ba8da6c22980f889d805a5300cbc5f69a8ef9f8b4ae13cf29f62e9d3ba"}
Nov 01 08:44:07.841871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:07.841763    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd" event={"ID":"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53","Type":"ContainerStarted","Data":"0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5"}
Nov 01 08:44:07.841871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:07.841870    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:07.843478 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:07.843453    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b" event={"ID":"1abaa356-6613-4f4d-8f41-d5bc02411535","Type":"ContainerStarted","Data":"2c01ce6b827a09cd7a74e7dce3e014d7b47b1624171eed0a5675580b66e2c55d"}
Nov 01 08:44:07.843611 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:07.843594    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:07.953762 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:07.953715    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd" podStartSLOduration=2.013610031 podStartE2EDuration="4.953695996s" totalImagesPullingTime="2.940085965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:07.952663695 +0000 UTC m=+718.067536662" watchObservedRunningTime="2026-11-01 08:44:07.953695996 +0000 UTC m=+718.068569007"
Nov 01 08:44:08.067548 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:08.067491    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b" podStartSLOduration=3.146971872 podStartE2EDuration="4.067477203s" totalImagesPullingTime="920.505331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:08.06547326 +0000 UTC m=+718.180346229" watchObservedRunningTime="2026-11-01 08:44:08.067477203 +0000 UTC m=+718.182350168"
Nov 01 08:44:13.854386 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:13.854353    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:44:15.414579 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.414547    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"]
Nov 01 08:44:15.419813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.419793    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.470539 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.470508    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f814ac10-serving-cert\""
Nov 01 08:44:15.498664 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.498612    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-openshift-service-ca\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.498874 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.498768    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.498874 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.498850    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-metrics-certs\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.499009 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.498892    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdq4\" (UniqueName: \"kubernetes.io/projected/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-kube-api-access-pkdq4\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.521953 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.521914    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"]
Nov 01 08:44:15.600515 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.600480    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.600515 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.600517    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-metrics-certs\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.600715 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.600541    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdq4\" (UniqueName: \"kubernetes.io/projected/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-kube-api-access-pkdq4\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.600715 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.600645    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-openshift-service-ca\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.600780 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:44:15.600647    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-serving-cert: secret "maas-api-e2e-migrate-f814ac10-serving-cert" not found
Nov 01 08:44:15.600828 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:44:15.600816    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls podName:7c55a5c0-e909-481e-9d1e-33c0c70ef5b3 nodeName:}" failed. No retries permitted until 2026-11-01 08:44:16.100795067 +0000 UTC m=+726.215668028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls") pod "maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7" (UID: "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3") : secret "maas-api-e2e-migrate-f814ac10-serving-cert" not found
Nov 01 08:44:15.601315 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.601282    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-openshift-service-ca\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.603138 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.603122    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-metrics-certs\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:15.663416 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:15.663389    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdq4\" (UniqueName: \"kubernetes.io/projected/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-kube-api-access-pkdq4\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:16.105554 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:16.105514    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:16.108096 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:16.108069    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls\") pod \"maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:16.334784 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:16.334747    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:16.704448 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:16.704415    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"]
Nov 01 08:44:16.707282 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:44:16.707240    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c55a5c0_e909_481e_9d1e_33c0c70ef5b3.slice/crio-f6a7dca8c1b625e2c2259fe56edba6d00136138b852081278b494650b6994614 WatchSource:0}: Error finding container f6a7dca8c1b625e2c2259fe56edba6d00136138b852081278b494650b6994614: Status 404 returned error can't find the container with id f6a7dca8c1b625e2c2259fe56edba6d00136138b852081278b494650b6994614
Nov 01 08:44:16.902876 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:16.902827    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7" event={"ID":"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3","Type":"ContainerStarted","Data":"f6a7dca8c1b625e2c2259fe56edba6d00136138b852081278b494650b6994614"}
Nov 01 08:44:17.911092 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:17.911055    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7" event={"ID":"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3","Type":"ContainerStarted","Data":"275ea19ece613593349d411882efc7b55ce8440106f4eb077207394a6ebe1d4d"}
Nov 01 08:44:17.911496 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:17.911165    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:18.025078 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:18.025009    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7" podStartSLOduration=2.145290036 podStartE2EDuration="3.024994203s" totalImagesPullingTime="879.704167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:18.02350865 +0000 UTC m=+728.138381619" watchObservedRunningTime="2026-11-01 08:44:18.024994203 +0000 UTC m=+728.139867170"
Nov 01 08:44:18.851740 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:18.851709    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:44:23.922121 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:23.922082    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:44:31.023245 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:44:31.023212    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:44:42.141631 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.141596    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"]
Nov 01 08:44:42.145661 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.145640    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.195747 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.195712    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0f765cd6-openshift-default-dockercfg-4fckl\""
Nov 01 08:44:42.245929 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.245899    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"]
Nov 01 08:44:42.253079 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253053    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.253209 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253087    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483b5b3-5141-4025-80ef-1311be2b2955-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.253251 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253205    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.253251 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253232    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483b5b3-5141-4025-80ef-1311be2b2955-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.253326 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253249    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.253326 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253290    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.253414 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253398    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq5jp\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-kube-api-access-lq5jp\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.253453 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253430    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.253492 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.253453    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.354448 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354413    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.354448 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354450    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.354698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354490    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.354698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354509    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483b5b3-5141-4025-80ef-1311be2b2955-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.354698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354549    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.354698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354573    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483b5b3-5141-4025-80ef-1311be2b2955-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.354698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354596    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.354698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354639    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.355013 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354715    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq5jp\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-kube-api-access-lq5jp\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.355013 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.354896    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.355154 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.355008    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.355199 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.355167    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.355459 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.355436    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.355508 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.355483    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483b5b3-5141-4025-80ef-1311be2b2955-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.356964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.356945    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.357240 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.357222    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483b5b3-5141-4025-80ef-1311be2b2955-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.413645 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.413568    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.413779 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.413747    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq5jp\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-kube-api-access-lq5jp\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.458284 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.458255    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:42.880760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.880735    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"]
Nov 01 08:44:42.882119 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:44:42.882092    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5483b5b3_5141_4025_80ef_1311be2b2955.slice/crio-8029becd35d5888a6709adeacff610a831dc9344295871cef1b3702cc60fe542 WatchSource:0}: Error finding container 8029becd35d5888a6709adeacff610a831dc9344295871cef1b3702cc60fe542: Status 404 returned error can't find the container with id 8029becd35d5888a6709adeacff610a831dc9344295871cef1b3702cc60fe542
Nov 01 08:44:42.884521 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.884490    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:44:42.884613 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.884558    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:44:42.884613 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:42.884585    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:44:43.056467 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:43.056428    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" event={"ID":"5483b5b3-5141-4025-80ef-1311be2b2955","Type":"ContainerStarted","Data":"1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8"}
Nov 01 08:44:43.056679 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:43.056506    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" event={"ID":"5483b5b3-5141-4025-80ef-1311be2b2955","Type":"ContainerStarted","Data":"8029becd35d5888a6709adeacff610a831dc9344295871cef1b3702cc60fe542"}
Nov 01 08:44:43.173985 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:43.173889    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" podStartSLOduration=1.173874504 podStartE2EDuration="1.173874504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:43.171776579 +0000 UTC m=+753.286649548" watchObservedRunningTime="2026-11-01 08:44:43.173874504 +0000 UTC m=+753.288747471"
Nov 01 08:44:43.459060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:43.458944    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:43.475064 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:43.474420    4931 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 08:44:43.475064 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:43.474490    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 08:44:44.481050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:44.474128    4931 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 08:44:44.481050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:44.474192    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 08:44:45.458860 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:45.458826    4931 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 08:44:45.459076 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:45.458879    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 08:44:46.458784 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:46.458735    4931 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 08:44:46.459357 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:46.458801    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 08:44:47.458925 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:47.458892    4931 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 08:44:47.459461 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:47.458951    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 08:44:48.477183 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:48.477144    4931 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 08:44:48.477797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:48.477765    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 08:44:49.463676 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:49.463598    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:49.463887 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:49.463869    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:44:49.464710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:44:49.464687    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:45:00.176700 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.176664    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"]
Nov 01 08:45:00.180521 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.180501    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:00.230718 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.230683    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 08:45:00.249183 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.249128    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x\" (UID: \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:00.249381 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.249217    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qvp\" (UniqueName: \"kubernetes.io/projected/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-kube-api-access-82qvp\") pod \"maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x\" (UID: \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:00.282437 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.282403    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"]
Nov 01 08:45:00.350287 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.350251    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qvp\" (UniqueName: \"kubernetes.io/projected/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-kube-api-access-82qvp\") pod \"maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x\" (UID: \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:00.350466 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.350391    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x\" (UID: \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:00.351069 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.351051    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x\" (UID: \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:00.409589 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.409549    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qvp\" (UniqueName: \"kubernetes.io/projected/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-kube-api-access-82qvp\") pod \"maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x\" (UID: \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:00.494240 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.494160    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:00.870697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:00.870663    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"]
Nov 01 08:45:00.872641 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:45:00.872611    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e7625_d680_4f7e_8ee8_a6eb7717abe4.slice/crio-cec4adbe10a12546b76942cca68079a6a4474a61f8519e2e39275597c5154835 WatchSource:0}: Error finding container cec4adbe10a12546b76942cca68079a6a4474a61f8519e2e39275597c5154835: Status 404 returned error can't find the container with id cec4adbe10a12546b76942cca68079a6a4474a61f8519e2e39275597c5154835
Nov 01 08:45:01.024424 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:01.024373    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:45:01.192666 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:01.192582    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x" event={"ID":"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4","Type":"ContainerStarted","Data":"cec4adbe10a12546b76942cca68079a6a4474a61f8519e2e39275597c5154835"}
Nov 01 08:45:01.960390 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:01.960353    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"]
Nov 01 08:45:01.960603 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:01.960558    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b" podUID="1abaa356-6613-4f4d-8f41-d5bc02411535" containerName="maas-api" containerID="cri-o://2c01ce6b827a09cd7a74e7dce3e014d7b47b1624171eed0a5675580b66e2c55d" gracePeriod=30
Nov 01 08:45:02.133994 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.133954    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"]
Nov 01 08:45:02.203501 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.203467    4931 generic.go:356] "Generic (PLEG): container finished" podID="1abaa356-6613-4f4d-8f41-d5bc02411535" containerID="2c01ce6b827a09cd7a74e7dce3e014d7b47b1624171eed0a5675580b66e2c55d" exitCode=0
Nov 01 08:45:02.203862 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.203543    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b" event={"ID":"1abaa356-6613-4f4d-8f41-d5bc02411535","Type":"ContainerDied","Data":"2c01ce6b827a09cd7a74e7dce3e014d7b47b1624171eed0a5675580b66e2c55d"}
Nov 01 08:45:02.241098 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.241066    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:45:02.387211 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.387179    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abaa356-6613-4f4d-8f41-d5bc02411535-kube-api-access-pgtrn\" (UniqueName: \"kubernetes.io/projected/1abaa356-6613-4f4d-8f41-d5bc02411535-kube-api-access-pgtrn\") pod \"1abaa356-6613-4f4d-8f41-d5bc02411535\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") "
Nov 01 08:45:02.387211 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.387214    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls\") pod \"1abaa356-6613-4f4d-8f41-d5bc02411535\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") "
Nov 01 08:45:02.387460 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.387329    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-metrics-certs\") pod \"1abaa356-6613-4f4d-8f41-d5bc02411535\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") "
Nov 01 08:45:02.387460 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.387372    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abaa356-6613-4f4d-8f41-d5bc02411535-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abaa356-6613-4f4d-8f41-d5bc02411535-openshift-service-ca\") pod \"1abaa356-6613-4f4d-8f41-d5bc02411535\" (UID: \"1abaa356-6613-4f4d-8f41-d5bc02411535\") "
Nov 01 08:45:02.387941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.387915    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abaa356-6613-4f4d-8f41-d5bc02411535-openshift-service-ca" pod "1abaa356-6613-4f4d-8f41-d5bc02411535" (UID: "1abaa356-6613-4f4d-8f41-d5bc02411535"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:02.390331 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.390310    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-metrics-certs" pod "1abaa356-6613-4f4d-8f41-d5bc02411535" (UID: "1abaa356-6613-4f4d-8f41-d5bc02411535"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:45:02.390427 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.390344    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls" pod "1abaa356-6613-4f4d-8f41-d5bc02411535" (UID: "1abaa356-6613-4f4d-8f41-d5bc02411535"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:45:02.390427 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.390366    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abaa356-6613-4f4d-8f41-d5bc02411535-kube-api-access-pgtrn" pod "1abaa356-6613-4f4d-8f41-d5bc02411535" (UID: "1abaa356-6613-4f4d-8f41-d5bc02411535"). InnerVolumeSpecName "kube-api-access-pgtrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:02.488437 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.488355    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:02.488437 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.488388    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abaa356-6613-4f4d-8f41-d5bc02411535-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:02.488437 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.488405    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgtrn\" (UniqueName: \"kubernetes.io/projected/1abaa356-6613-4f4d-8f41-d5bc02411535-kube-api-access-pgtrn\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:02.488437 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:02.488419    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abaa356-6613-4f4d-8f41-d5bc02411535-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:03.212146 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:03.212119    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"
Nov 01 08:45:03.212685 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:03.212117    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b" event={"ID":"1abaa356-6613-4f4d-8f41-d5bc02411535","Type":"ContainerDied","Data":"20fdf7ba8da6c22980f889d805a5300cbc5f69a8ef9f8b4ae13cf29f62e9d3ba"}
Nov 01 08:45:03.212788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:03.212730    4931 scope.go:122] "RemoveContainer" containerID="2c01ce6b827a09cd7a74e7dce3e014d7b47b1624171eed0a5675580b66e2c55d"
Nov 01 08:45:03.375723 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:03.375688    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"]
Nov 01 08:45:03.380798 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:03.380768    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c25ae7a-649794574b-f2j6b"]
Nov 01 08:45:04.049978 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:04.049935    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abaa356-6613-4f4d-8f41-d5bc02411535" path="/var/lib/kubelet/pods/1abaa356-6613-4f4d-8f41-d5bc02411535/volumes"
Nov 01 08:45:05.227965 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.227874    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x" event={"ID":"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4","Type":"ContainerStarted","Data":"ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1"}
Nov 01 08:45:05.227965 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.227923    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x" podUID="ab4e7625-d680-4f7e-8ee8-a6eb7717abe4" containerName="cleanup" containerID="cri-o://ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1" gracePeriod=30
Nov 01 08:45:05.340390 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.340343    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x" podStartSLOduration=1.498070517 podStartE2EDuration="5.340329937s" totalImagesPullingTime="3.84225942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:05.33923794 +0000 UTC m=+775.454110907" watchObservedRunningTime="2026-11-01 08:45:05.340329937 +0000 UTC m=+775.455202905"
Nov 01 08:45:05.677476 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.677449    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:05.834872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.834772    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-maas-api-service-ca\") pod \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\" (UID: \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\") "
Nov 01 08:45:05.835077 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.834873    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-kube-api-access-82qvp\" (UniqueName: \"kubernetes.io/projected/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-kube-api-access-82qvp\") pod \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\" (UID: \"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4\") "
Nov 01 08:45:05.835206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.835176    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-maas-api-service-ca" pod "ab4e7625-d680-4f7e-8ee8-a6eb7717abe4" (UID: "ab4e7625-d680-4f7e-8ee8-a6eb7717abe4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:05.835546 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.835526    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:05.837325 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.837296    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-kube-api-access-82qvp" pod "ab4e7625-d680-4f7e-8ee8-a6eb7717abe4" (UID: "ab4e7625-d680-4f7e-8ee8-a6eb7717abe4"). InnerVolumeSpecName "kube-api-access-82qvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:05.937780 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:05.937739    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qvp\" (UniqueName: \"kubernetes.io/projected/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4-kube-api-access-82qvp\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:06.235121 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.235078    4931 generic.go:356] "Generic (PLEG): container finished" podID="ab4e7625-d680-4f7e-8ee8-a6eb7717abe4" containerID="ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1" exitCode=6
Nov 01 08:45:06.235580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.235129    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"
Nov 01 08:45:06.235580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.235167    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x" event={"ID":"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4","Type":"ContainerDied","Data":"ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1"}
Nov 01 08:45:06.235580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.235222    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x" event={"ID":"ab4e7625-d680-4f7e-8ee8-a6eb7717abe4","Type":"ContainerDied","Data":"cec4adbe10a12546b76942cca68079a6a4474a61f8519e2e39275597c5154835"}
Nov 01 08:45:06.235580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.235244    4931 scope.go:122] "RemoveContainer" containerID="ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1"
Nov 01 08:45:06.245394 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.245375    4931 scope.go:122] "RemoveContainer" containerID="ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1"
Nov 01 08:45:06.245707 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:06.245681    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1\": container with ID starting with ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1 not found: ID does not exist" containerID="ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1"
Nov 01 08:45:06.245830 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.245713    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1"} err="failed to get container status \"ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1\": rpc error: code = NotFound desc = could not find container \"ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1\": container with ID starting with ff7f784ed1b115580206d45ac72a217bfae287e1f7b3aaa32c35104069db70e1 not found: ID does not exist"
Nov 01 08:45:06.395701 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.395664    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"]
Nov 01 08:45:06.398335 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:06.398311    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5c25ae7a-29892045-qtz4x"]
Nov 01 08:45:07.478180 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:07.478141    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"]
Nov 01 08:45:08.047475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:08.047442    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4e7625-d680-4f7e-8ee8-a6eb7717abe4" path="/var/lib/kubelet/pods/ab4e7625-d680-4f7e-8ee8-a6eb7717abe4/volumes"
Nov 01 08:45:12.479447 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:12.479377    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd" podUID="cbcf7dc5-d8a6-4178-9d15-a18442dd6d53" containerName="payload-processing" containerID="cri-o://0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5" gracePeriod=25
Nov 01 08:45:12.756546 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:12.756519    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:45:12.801763 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:12.801722    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-kube-api-access-fp294\" (UniqueName: \"kubernetes.io/projected/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-kube-api-access-fp294\") pod \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\" (UID: \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\") "
Nov 01 08:45:12.802176 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:12.802136    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-plugins-config-volume\") pod \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\" (UID: \"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53\") "
Nov 01 08:45:12.803100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:12.802838    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-plugins-config-volume" pod "cbcf7dc5-d8a6-4178-9d15-a18442dd6d53" (UID: "cbcf7dc5-d8a6-4178-9d15-a18442dd6d53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:12.803678 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:12.803659    4931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-plugins-config-volume\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:12.810302 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:12.810256    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-kube-api-access-fp294" pod "cbcf7dc5-d8a6-4178-9d15-a18442dd6d53" (UID: "cbcf7dc5-d8a6-4178-9d15-a18442dd6d53"). InnerVolumeSpecName "kube-api-access-fp294". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:12.909916 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:12.909873    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp294\" (UniqueName: \"kubernetes.io/projected/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53-kube-api-access-fp294\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:13.285427 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.285385    4931 generic.go:356] "Generic (PLEG): container finished" podID="cbcf7dc5-d8a6-4178-9d15-a18442dd6d53" containerID="0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5" exitCode=0
Nov 01 08:45:13.285711 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.285437    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"
Nov 01 08:45:13.285711 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.285448    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd" event={"ID":"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53","Type":"ContainerDied","Data":"0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5"}
Nov 01 08:45:13.285711 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.285528    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd" event={"ID":"cbcf7dc5-d8a6-4178-9d15-a18442dd6d53","Type":"ContainerDied","Data":"303cfb26203f41d2d3f9610ff7eb9e9c2868be77f1b9f9be37341e518255d16a"}
Nov 01 08:45:13.285711 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.285556    4931 scope.go:122] "RemoveContainer" containerID="0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5"
Nov 01 08:45:13.295184 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.295157    4931 scope.go:122] "RemoveContainer" containerID="0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5"
Nov 01 08:45:13.295482 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:13.295464    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5\": container with ID starting with 0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5 not found: ID does not exist" containerID="0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5"
Nov 01 08:45:13.295530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.295492    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5"} err="failed to get container status \"0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5\": rpc error: code = NotFound desc = could not find container \"0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5\": container with ID starting with 0dd59b01ed7d71905e9cdb1bdfce8e34f853ea3d16790ebe7e5589f85bc520b5 not found: ID does not exist"
Nov 01 08:45:13.443091 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.443055    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"]
Nov 01 08:45:13.490891 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:13.490848    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f814ac10-5cf4966ff7-wjsgd"]
Nov 01 08:45:14.046967 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:14.046930    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcf7dc5-d8a6-4178-9d15-a18442dd6d53" path="/var/lib/kubelet/pods/cbcf7dc5-d8a6-4178-9d15-a18442dd6d53/volumes"
Nov 01 08:45:20.200790 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.200759    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"]
Nov 01 08:45:20.201280 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.200961    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7" podUID="7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" containerName="maas-api" containerID="cri-o://275ea19ece613593349d411882efc7b55ce8440106f4eb077207394a6ebe1d4d" gracePeriod=30
Nov 01 08:45:20.342488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.342451    4931 generic.go:356] "Generic (PLEG): container finished" podID="7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" containerID="275ea19ece613593349d411882efc7b55ce8440106f4eb077207394a6ebe1d4d" exitCode=0
Nov 01 08:45:20.342658 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.342503    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7" event={"ID":"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3","Type":"ContainerDied","Data":"275ea19ece613593349d411882efc7b55ce8440106f4eb077207394a6ebe1d4d"}
Nov 01 08:45:20.493246 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.493219    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:45:20.599627 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.599601    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls\") pod \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") "
Nov 01 08:45:20.599807 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.599636    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-metrics-certs\") pod \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") "
Nov 01 08:45:20.599807 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.599678    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-kube-api-access-pkdq4\" (UniqueName: \"kubernetes.io/projected/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-kube-api-access-pkdq4\") pod \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") "
Nov 01 08:45:20.599807 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.599753    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-openshift-service-ca\") pod \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\" (UID: \"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3\") "
Nov 01 08:45:20.600192 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.600165    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-openshift-service-ca" pod "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" (UID: "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:20.602081 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.602054    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-metrics-certs" pod "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" (UID: "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:45:20.602081 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.602072    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-kube-api-access-pkdq4" pod "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" (UID: "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3"). InnerVolumeSpecName "kube-api-access-pkdq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:20.602230 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.602084    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls" pod "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" (UID: "7c55a5c0-e909-481e-9d1e-33c0c70ef5b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:45:20.701407 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.701367    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:20.701407 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.701396    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:20.701407 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.701406    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:20.701407 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:20.701415    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkdq4\" (UniqueName: \"kubernetes.io/projected/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3-kube-api-access-pkdq4\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:21.349332 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:21.349301    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"
Nov 01 08:45:21.349800 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:21.349339    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7" event={"ID":"7c55a5c0-e909-481e-9d1e-33c0c70ef5b3","Type":"ContainerDied","Data":"f6a7dca8c1b625e2c2259fe56edba6d00136138b852081278b494650b6994614"}
Nov 01 08:45:21.349800 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:21.349399    4931 scope.go:122] "RemoveContainer" containerID="275ea19ece613593349d411882efc7b55ce8440106f4eb077207394a6ebe1d4d"
Nov 01 08:45:21.507739 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:21.507704    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"]
Nov 01 08:45:21.555509 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:21.555473    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f814ac10-645dcb5488-74dl7"]
Nov 01 08:45:22.046936 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:22.046902    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" path="/var/lib/kubelet/pods/7c55a5c0-e909-481e-9d1e-33c0c70ef5b3/volumes"
Nov 01 08:45:27.330143 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330108    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"]
Nov 01 08:45:27.330540 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330527    4931 cpu_manager.go:414] "containerMap: removing container" podUID="ab4e7625-d680-4f7e-8ee8-a6eb7717abe4" containerName="cleanup"
Nov 01 08:45:27.330583 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330541    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4e7625-d680-4f7e-8ee8-a6eb7717abe4" containerName="cleanup"
Nov 01 08:45:27.330583 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330553    4931 cpu_manager.go:414] "containerMap: removing container" podUID="7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" containerName="maas-api"
Nov 01 08:45:27.330583 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330558    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" containerName="maas-api"
Nov 01 08:45:27.330583 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330573    4931 cpu_manager.go:414] "containerMap: removing container" podUID="1abaa356-6613-4f4d-8f41-d5bc02411535" containerName="maas-api"
Nov 01 08:45:27.330583 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330579    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abaa356-6613-4f4d-8f41-d5bc02411535" containerName="maas-api"
Nov 01 08:45:27.330735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330593    4931 cpu_manager.go:414] "containerMap: removing container" podUID="cbcf7dc5-d8a6-4178-9d15-a18442dd6d53" containerName="payload-processing"
Nov 01 08:45:27.330735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330598    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcf7dc5-d8a6-4178-9d15-a18442dd6d53" containerName="payload-processing"
Nov 01 08:45:27.330735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330664    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcf7dc5-d8a6-4178-9d15-a18442dd6d53" containerName="payload-processing"
Nov 01 08:45:27.330735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330675    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4e7625-d680-4f7e-8ee8-a6eb7717abe4" containerName="cleanup"
Nov 01 08:45:27.330735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330682    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c55a5c0-e909-481e-9d1e-33c0c70ef5b3" containerName="maas-api"
Nov 01 08:45:27.330735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.330689    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abaa356-6613-4f4d-8f41-d5bc02411535" containerName="maas-api"
Nov 01 08:45:27.336137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.336118    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.434052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.433999    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-751a73-serving-cert\""
Nov 01 08:45:27.436328 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.436304    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"]
Nov 01 08:45:27.466918 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.466884    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4282f02f-5630-4392-94c3-7f83cdcecc1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.467226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.467194    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.467346 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.467272    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-metrics-certs\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.467346 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.467320    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzw4l\" (UniqueName: \"kubernetes.io/projected/4282f02f-5630-4392-94c3-7f83cdcecc1a-kube-api-access-nzw4l\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.568291 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.568252    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-metrics-certs\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.568462 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.568320    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzw4l\" (UniqueName: \"kubernetes.io/projected/4282f02f-5630-4392-94c3-7f83cdcecc1a-kube-api-access-nzw4l\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.568462 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.568381    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4282f02f-5630-4392-94c3-7f83cdcecc1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.568462 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.568451    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.568868 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:27.568568    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-serving-cert: secret "maas-api-e2e-worker-w0-751a73-serving-cert" not found
Nov 01 08:45:27.568868 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:27.568649    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls podName:4282f02f-5630-4392-94c3-7f83cdcecc1a nodeName:}" failed. No retries permitted until 2026-11-01 08:45:28.06862794 +0000 UTC m=+798.183500900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls") pod "maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt" (UID: "4282f02f-5630-4392-94c3-7f83cdcecc1a") : secret "maas-api-e2e-worker-w0-751a73-serving-cert" not found
Nov 01 08:45:27.569563 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.569540    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4282f02f-5630-4392-94c3-7f83cdcecc1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.571180 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.571155    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-metrics-certs\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:27.629716 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:27.629686    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzw4l\" (UniqueName: \"kubernetes.io/projected/4282f02f-5630-4392-94c3-7f83cdcecc1a-kube-api-access-nzw4l\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:28.073072 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:28.072975    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:28.075513 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:28.075495    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:28.249784 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:28.249749    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:28.618190 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:28.618156    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"]
Nov 01 08:45:28.621181 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:45:28.621143    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4282f02f_5630_4392_94c3_7f83cdcecc1a.slice/crio-a8e88ee6f71a6257c34b68f60b1e8f4bd50c531efcc37bd908d158637648dab6 WatchSource:0}: Error finding container a8e88ee6f71a6257c34b68f60b1e8f4bd50c531efcc37bd908d158637648dab6: Status 404 returned error can't find the container with id a8e88ee6f71a6257c34b68f60b1e8f4bd50c531efcc37bd908d158637648dab6
Nov 01 08:45:29.400094 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:29.400062    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt" event={"ID":"4282f02f-5630-4392-94c3-7f83cdcecc1a","Type":"ContainerStarted","Data":"a8e88ee6f71a6257c34b68f60b1e8f4bd50c531efcc37bd908d158637648dab6"}
Nov 01 08:45:30.407825 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:30.407785    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt" event={"ID":"4282f02f-5630-4392-94c3-7f83cdcecc1a","Type":"ContainerStarted","Data":"4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826"}
Nov 01 08:45:30.408409 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:30.408388    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:30.522350 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:30.522295    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt" podStartSLOduration=2.762936061 podStartE2EDuration="3.522281559s" totalImagesPullingTime="759.345498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:30.521240146 +0000 UTC m=+800.636113114" watchObservedRunningTime="2026-11-01 08:45:30.522281559 +0000 UTC m=+800.637154527"
Nov 01 08:45:31.021836 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:31.021803    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:45:36.419375 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:36.419342    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 08:45:37.355580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.355547    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"]
Nov 01 08:45:37.359316 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.359297    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.411991 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.411948    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b711ae-serving-cert\""
Nov 01 08:45:37.461710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.461680    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"]
Nov 01 08:45:37.469139 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.469116    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.469301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.469171    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2rs\" (UniqueName: \"kubernetes.io/projected/4b00f5ed-abe0-4621-a8a0-530838e68b7c-kube-api-access-2j2rs\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.469301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.469231    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-metrics-certs\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.469301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.469275    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b00f5ed-abe0-4621-a8a0-530838e68b7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.569793 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.569755    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-metrics-certs\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.569971 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.569802    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b00f5ed-abe0-4621-a8a0-530838e68b7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.569971 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.569880    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.569971 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.569926    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2rs\" (UniqueName: \"kubernetes.io/projected/4b00f5ed-abe0-4621-a8a0-530838e68b7c-kube-api-access-2j2rs\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.570187 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:37.570074    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-serving-cert: secret "maas-api-e2e-worker-w4-b711ae-serving-cert" not found
Nov 01 08:45:37.570187 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:37.570153    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls podName:4b00f5ed-abe0-4621-a8a0-530838e68b7c nodeName:}" failed. No retries permitted until 2026-11-01 08:45:38.070130635 +0000 UTC m=+808.185003584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls") pod "maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h" (UID: "4b00f5ed-abe0-4621-a8a0-530838e68b7c") : secret "maas-api-e2e-worker-w4-b711ae-serving-cert" not found
Nov 01 08:45:37.570506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.570485    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b00f5ed-abe0-4621-a8a0-530838e68b7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.572407 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.572382    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-metrics-certs\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:37.629300 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:37.629224    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2rs\" (UniqueName: \"kubernetes.io/projected/4b00f5ed-abe0-4621-a8a0-530838e68b7c-kube-api-access-2j2rs\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:38.076109 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:38.076003    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:38.078516 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:38.078495    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:38.272753 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:38.272717    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:38.640205 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:38.640172    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"]
Nov 01 08:45:38.641295 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:45:38.641269    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b00f5ed_abe0_4621_a8a0_530838e68b7c.slice/crio-fd8b3a801ee95b7df6f491487b6048255e7985abb1466706f4f2997b592d0eb0 WatchSource:0}: Error finding container fd8b3a801ee95b7df6f491487b6048255e7985abb1466706f4f2997b592d0eb0: Status 404 returned error can't find the container with id fd8b3a801ee95b7df6f491487b6048255e7985abb1466706f4f2997b592d0eb0
Nov 01 08:45:39.458899 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:39.458863    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h" event={"ID":"4b00f5ed-abe0-4621-a8a0-530838e68b7c","Type":"ContainerStarted","Data":"fd8b3a801ee95b7df6f491487b6048255e7985abb1466706f4f2997b592d0eb0"}
Nov 01 08:45:40.465855 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:40.465813    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h" event={"ID":"4b00f5ed-abe0-4621-a8a0-530838e68b7c","Type":"ContainerStarted","Data":"c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a"}
Nov 01 08:45:40.466334 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:40.466014    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:40.580384 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:40.580322    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h" podStartSLOduration=2.615366185 podStartE2EDuration="3.580303556s" totalImagesPullingTime="964.937371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:40.578566655 +0000 UTC m=+810.693439624" watchObservedRunningTime="2026-11-01 08:45:40.580303556 +0000 UTC m=+810.695176526"
Nov 01 08:45:44.048433 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:44.048399    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"]
Nov 01 08:45:44.049114 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:44.048592    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy" containerID="cri-o://1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8" gracePeriod=30
Nov 01 08:45:46.477530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:46.477495    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 08:45:48.438200 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.438166    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"]
Nov 01 08:45:48.441755 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.441734    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.494901 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.494863    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-54e01b-serving-cert\""
Nov 01 08:45:48.544883 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.544846    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"]
Nov 01 08:45:48.575838 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.575808    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-metrics-certs\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.575838 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.575840    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.576059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.575934    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jd4\" (UniqueName: \"kubernetes.io/projected/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-kube-api-access-k5jd4\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.576059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.576001    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.677541 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.677509    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.677710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.677573    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jd4\" (UniqueName: \"kubernetes.io/projected/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-kube-api-access-k5jd4\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.677710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.677610    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.677710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.677695    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-metrics-certs\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.677828 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:48.677800    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-serving-cert: secret "maas-api-e2e-worker-w3-54e01b-serving-cert" not found
Nov 01 08:45:48.677933 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:48.677908    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls podName:e6ebc602-27d4-4f59-b8d0-f5a99d58b973 nodeName:}" failed. No retries permitted until 2026-11-01 08:45:49.177885697 +0000 UTC m=+819.292758647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls") pod "maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl" (UID: "e6ebc602-27d4-4f59-b8d0-f5a99d58b973") : secret "maas-api-e2e-worker-w3-54e01b-serving-cert" not found
Nov 01 08:45:48.678336 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.678318    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.680266 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.680245    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-metrics-certs\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:48.736795 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:48.736734    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jd4\" (UniqueName: \"kubernetes.io/projected/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-kube-api-access-k5jd4\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:49.183736 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.183701    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:49.186120 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.186098    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls\") pod \"maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:49.309169 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.309148    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:45:49.355211 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.355179    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:49.486972 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.486939    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5483b5b3-5141-4025-80ef-1311be2b2955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483b5b3-5141-4025-80ef-1311be2b2955-istiod-ca-cert\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.486972 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.486975    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-socket\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487000    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-data\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487058    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-istio-token\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-istio-token\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487095    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-kube-api-access-lq5jp\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-kube-api-access-lq5jp\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487117    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-credential-socket\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487134    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5483b5b3-5141-4025-80ef-1311be2b2955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483b5b3-5141-4025-80ef-1311be2b2955-istio-podinfo\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487173    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-certs\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487271    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-socket" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487311    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-envoy\") pod \"5483b5b3-5141-4025-80ef-1311be2b2955\" (UID: \"5483b5b3-5141-4025-80ef-1311be2b2955\") "
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487339    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-data" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.487488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487429    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5483b5b3-5141-4025-80ef-1311be2b2955-istiod-ca-cert" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:45:49.488201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487572    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-certs" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.488201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487596    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-credential-socket" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.488201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487732    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.488201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487755    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.488201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487776    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483b5b3-5141-4025-80ef-1311be2b2955-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.488201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487790    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.488201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.487804    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.489641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.489615    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5483b5b3-5141-4025-80ef-1311be2b2955-istio-podinfo" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 08:45:49.489641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.489631    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-istio-token" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:49.489786 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.489644    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-envoy" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:45:49.489786 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.489618    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-kube-api-access-lq5jp" pod "5483b5b3-5141-4025-80ef-1311be2b2955" (UID: "5483b5b3-5141-4025-80ef-1311be2b2955"). InnerVolumeSpecName "kube-api-access-lq5jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:45:49.520823 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.520794    4931 generic.go:356] "Generic (PLEG): container finished" podID="5483b5b3-5141-4025-80ef-1311be2b2955" containerID="1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8" exitCode=0
Nov 01 08:45:49.520991 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.520838    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"
Nov 01 08:45:49.520991 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.520859    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" event={"ID":"5483b5b3-5141-4025-80ef-1311be2b2955","Type":"ContainerDied","Data":"1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8"}
Nov 01 08:45:49.520991 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.520915    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb" event={"ID":"5483b5b3-5141-4025-80ef-1311be2b2955","Type":"ContainerDied","Data":"8029becd35d5888a6709adeacff610a831dc9344295871cef1b3702cc60fe542"}
Nov 01 08:45:49.520991 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.520944    4931 scope.go:122] "RemoveContainer" containerID="1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8"
Nov 01 08:45:49.532314 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.532283    4931 scope.go:122] "RemoveContainer" containerID="1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8"
Nov 01 08:45:49.532595 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:45:49.532577    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8\": container with ID starting with 1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8 not found: ID does not exist" containerID="1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8"
Nov 01 08:45:49.532657 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.532604    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8"} err="failed to get container status \"1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8\": rpc error: code = NotFound desc = could not find container \"1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8\": container with ID starting with 1dd846d7d1ee16d4a1c75411a97292c02f9eaadda5d521b8ba1dea908cfb3ed8 not found: ID does not exist"
Nov 01 08:45:49.588868 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.588822    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.588868 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.588867    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq5jp\" (UniqueName: \"kubernetes.io/projected/5483b5b3-5141-4025-80ef-1311be2b2955-kube-api-access-lq5jp\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.589158 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.588883    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483b5b3-5141-4025-80ef-1311be2b2955-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.589158 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.588896    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483b5b3-5141-4025-80ef-1311be2b2955-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:45:49.679634 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.679553    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"]
Nov 01 08:45:49.685427 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.685396    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-bsmrb"]
Nov 01 08:45:49.726539 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:49.726514    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"]
Nov 01 08:45:49.727894 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:45:49.727865    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ebc602_27d4_4f59_b8d0_f5a99d58b973.slice/crio-c3a31a66f0c8512e63c219ad2e51ce52b35e828a11716c2a10b99ecddc842dfa WatchSource:0}: Error finding container c3a31a66f0c8512e63c219ad2e51ce52b35e828a11716c2a10b99ecddc842dfa: Status 404 returned error can't find the container with id c3a31a66f0c8512e63c219ad2e51ce52b35e828a11716c2a10b99ecddc842dfa
Nov 01 08:45:50.047212 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:50.047140    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" path="/var/lib/kubelet/pods/5483b5b3-5141-4025-80ef-1311be2b2955/volumes"
Nov 01 08:45:50.527710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:50.527673    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl" event={"ID":"e6ebc602-27d4-4f59-b8d0-f5a99d58b973","Type":"ContainerStarted","Data":"c3a31a66f0c8512e63c219ad2e51ce52b35e828a11716c2a10b99ecddc842dfa"}
Nov 01 08:45:51.536460 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:51.536424    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl" event={"ID":"e6ebc602-27d4-4f59-b8d0-f5a99d58b973","Type":"ContainerStarted","Data":"22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0"}
Nov 01 08:45:51.536904 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:51.536497    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:45:51.649766 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:51.649712    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl" podStartSLOduration=2.84417184 podStartE2EDuration="3.649698592s" totalImagesPullingTime="805.526752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:51.649256558 +0000 UTC m=+821.764129527" watchObservedRunningTime="2026-11-01 08:45:51.649698592 +0000 UTC m=+821.764571557"
Nov 01 08:45:57.547834 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:45:57.547807    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 08:46:01.022638 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:46:01.022600    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:46:20.143861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.143775    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"]
Nov 01 08:46:20.144475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.144209    4931 cpu_manager.go:414] "containerMap: removing container" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy"
Nov 01 08:46:20.144475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.144222    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy"
Nov 01 08:46:20.144475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.144357    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5483b5b3-5141-4025-80ef-1311be2b2955" containerName="istio-proxy"
Nov 01 08:46:20.149363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.149342    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.215597 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.215573    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"kube-root-ca.crt\""
Nov 01 08:46:20.252397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.252371    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"default-dockercfg-t4wrj\""
Nov 01 08:46:20.252397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.252372    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"openshift-service-ca.crt\""
Nov 01 08:46:20.252575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.252545    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"simulator-w0-751a73-kserve-self-signed-certs\""
Nov 01 08:46:20.254844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.254824    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"]
Nov 01 08:46:20.273000 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.272969    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nw8\" (UniqueName: \"kubernetes.io/projected/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-kube-api-access-55nw8\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.273166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.273119    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-model-cache\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.273166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.273154    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tmp-dir\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.273290 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.273263    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tls-certs\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.273363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.273351    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-dshm\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.273449 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.273435    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-home\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.374754 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.374719    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-dshm\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.374754 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.374770    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-home\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.375051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.374796    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nw8\" (UniqueName: \"kubernetes.io/projected/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-kube-api-access-55nw8\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.375051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.374834    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-model-cache\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.375051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.374860    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tmp-dir\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.375051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.374899    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tls-certs\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.375275 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.375254    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-home\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.375622 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.375544    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-model-cache\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.375622 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.375566    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tmp-dir\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.377575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.377548    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tls-certs\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.377685 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.377605    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-dshm\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.434311 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.434236    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nw8\" (UniqueName: \"kubernetes.io/projected/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-kube-api-access-55nw8\") pod \"simulator-w0-751a73-kserve-7bf9b5679f-xcwb5\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") " pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.485547 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.485510    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:20.854388 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:20.854363    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"]
Nov 01 08:46:20.855297 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:46:20.855271    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41d7b4cb_e2d0_4f0c_a1e6_0b4a887acdcc.slice/crio-d7d183ad88f93435633b6db0088756cfc48f0fec5dc9c99d39d74117e379d958 WatchSource:0}: Error finding container d7d183ad88f93435633b6db0088756cfc48f0fec5dc9c99d39d74117e379d958: Status 404 returned error can't find the container with id d7d183ad88f93435633b6db0088756cfc48f0fec5dc9c99d39d74117e379d958
Nov 01 08:46:21.707020 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:21.706979    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" event={"ID":"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc","Type":"ContainerStarted","Data":"d7d183ad88f93435633b6db0088756cfc48f0fec5dc9c99d39d74117e379d958"}
Nov 01 08:46:23.720877 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:23.720825    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" event={"ID":"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc","Type":"ContainerStarted","Data":"d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb"}
Nov 01 08:46:23.836583 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:23.836529    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podStartSLOduration=1.701360004 podStartE2EDuration="3.836514505s" totalImagesPullingTime="2.135154501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:46:23.83484117 +0000 UTC m=+853.949714138" watchObservedRunningTime="2026-11-01 08:46:23.836514505 +0000 UTC m=+853.951387548"
Nov 01 08:46:24.485831 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:24.485788    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:25.429844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.429811    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"]
Nov 01 08:46:25.433962 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.433937    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.488700 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.488671    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"]
Nov 01 08:46:25.526127 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.526090    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tls-certs\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.526315 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.526135    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2m6m\" (UniqueName: \"kubernetes.io/projected/cb9d0733-3c83-462e-bbd3-c2be6acd5327-kube-api-access-n2m6m\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.526315 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.526250    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-model-cache\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.526440 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.526321    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-dshm\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.526440 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.526354    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-home\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.526440 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.526410    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tmp-dir\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.533789 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.533754    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"kube-root-ca.crt\""
Nov 01 08:46:25.533941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.533821    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"openshift-service-ca.crt\""
Nov 01 08:46:25.533941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.533881    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"default-dockercfg-q6wkx\""
Nov 01 08:46:25.533941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.533918    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"simulator-w4-b711ae-kserve-self-signed-certs\""
Nov 01 08:46:25.627127 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627098    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tls-certs\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.627323 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627140    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2m6m\" (UniqueName: \"kubernetes.io/projected/cb9d0733-3c83-462e-bbd3-c2be6acd5327-kube-api-access-n2m6m\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.627323 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627163    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-model-cache\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.627323 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627222    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-dshm\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.627323 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627284    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-home\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.627323 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627307    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tmp-dir\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.627822 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627788    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tmp-dir\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.627945 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627841    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-home\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.627945 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.627880    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-model-cache\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.629793 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.629774    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-dshm\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.630190 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.630171    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tls-certs\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.686436 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.686347    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2m6m\" (UniqueName: \"kubernetes.io/projected/cb9d0733-3c83-462e-bbd3-c2be6acd5327-kube-api-access-n2m6m\") pod \"simulator-w4-b711ae-kserve-549dc8946-wjwhz\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") " pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:25.748907 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:25.748876    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:26.126900 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:26.126856    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"]
Nov 01 08:46:26.128513 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:46:26.128479    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb9d0733_3c83_462e_bbd3_c2be6acd5327.slice/crio-57c41652dedaabe7f73a0118ecb4f3612e205610bb861d843309681ad0ced549 WatchSource:0}: Error finding container 57c41652dedaabe7f73a0118ecb4f3612e205610bb861d843309681ad0ced549: Status 404 returned error can't find the container with id 57c41652dedaabe7f73a0118ecb4f3612e205610bb861d843309681ad0ced549
Nov 01 08:46:26.740188 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:26.740151    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" event={"ID":"cb9d0733-3c83-462e-bbd3-c2be6acd5327","Type":"ContainerStarted","Data":"b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49"}
Nov 01 08:46:26.740656 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:26.740198    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" event={"ID":"cb9d0733-3c83-462e-bbd3-c2be6acd5327","Type":"ContainerStarted","Data":"57c41652dedaabe7f73a0118ecb4f3612e205610bb861d843309681ad0ced549"}
Nov 01 08:46:26.749533 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:26.749503    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:26.853369 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:26.853311    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podStartSLOduration=1.853295418 podStartE2EDuration="1.853295418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:46:26.85121487 +0000 UTC m=+856.966087838" watchObservedRunningTime="2026-11-01 08:46:26.853295418 +0000 UTC m=+856.968168386"
Nov 01 08:46:30.486293 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:30.486255    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:30.499168 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:30.499133    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:30.775554 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:30.775471    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:46:31.024292 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:46:31.024251    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:46:35.749359 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:35.749327    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:35.762741 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:35.762715    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:35.805140 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:35.805109    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:46:57.944612 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:57.944578    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"]
Nov 01 08:46:57.951054 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:57.951018    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.004384 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.004351    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"]
Nov 01 08:46:58.048371 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.048343    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"kube-root-ca.crt\""
Nov 01 08:46:58.048371 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.048368    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"openshift-service-ca.crt\""
Nov 01 08:46:58.049809 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.049792    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-model-cache\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.049873 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.049837    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd69395-69b6-443c-8023-9a83841d9aa6-tls-certs\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.049974 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.049954    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-tmp-dir\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.050053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.050012    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-dshm\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.050104 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.050076    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9j8v\" (UniqueName: \"kubernetes.io/projected/1fd69395-69b6-443c-8023-9a83841d9aa6-kube-api-access-q9j8v\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.050174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.050162    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-home\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.096201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.096176    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"simulator-w3-54e01b-kserve-self-signed-certs\""
Nov 01 08:46:58.121698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.121667    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"default-dockercfg-fp8xg\""
Nov 01 08:46:58.151260 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.151224    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-home\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.151453 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.151275    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-model-cache\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.151453 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.151426    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd69395-69b6-443c-8023-9a83841d9aa6-tls-certs\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.151581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.151486    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-tmp-dir\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.151581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.151534    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-dshm\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.151581 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.151563    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9j8v\" (UniqueName: \"kubernetes.io/projected/1fd69395-69b6-443c-8023-9a83841d9aa6-kube-api-access-q9j8v\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.151762 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.151744    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-model-cache\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.151841 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.151801    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-home\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.152250 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.152212    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-tmp-dir\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.154114 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.154089    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-dshm\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.155080 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.155062    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd69395-69b6-443c-8023-9a83841d9aa6-tls-certs\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.210661 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.210576    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9j8v\" (UniqueName: \"kubernetes.io/projected/1fd69395-69b6-443c-8023-9a83841d9aa6-kube-api-access-q9j8v\") pod \"simulator-w3-54e01b-kserve-d5c956f75-qq55j\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") " pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.265492 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.265447    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:46:58.640190 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.640164    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"]
Nov 01 08:46:58.641577 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:46:58.641547    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fd69395_69b6_443c_8023_9a83841d9aa6.slice/crio-35c84f0000076410ea64d1464ccdaf5d093c6fcba47a3ad7af3ed1b895ee1da9 WatchSource:0}: Error finding container 35c84f0000076410ea64d1464ccdaf5d093c6fcba47a3ad7af3ed1b895ee1da9: Status 404 returned error can't find the container with id 35c84f0000076410ea64d1464ccdaf5d093c6fcba47a3ad7af3ed1b895ee1da9
Nov 01 08:46:58.923043 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.922928    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" event={"ID":"1fd69395-69b6-443c-8023-9a83841d9aa6","Type":"ContainerStarted","Data":"65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3"}
Nov 01 08:46:58.923043 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:58.922976    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" event={"ID":"1fd69395-69b6-443c-8023-9a83841d9aa6","Type":"ContainerStarted","Data":"35c84f0000076410ea64d1464ccdaf5d093c6fcba47a3ad7af3ed1b895ee1da9"}
Nov 01 08:46:59.035768 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:59.035717    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podStartSLOduration=2.035702895 podStartE2EDuration="2.035702895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:46:59.034118704 +0000 UTC m=+889.148991673" watchObservedRunningTime="2026-11-01 08:46:59.035702895 +0000 UTC m=+889.150575859"
Nov 01 08:46:59.265974 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:46:59.265894    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:47:01.023850 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:47:01.023814    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:47:08.266332 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:08.266298    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:47:08.280469 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:08.280443    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:47:08.993996 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:08.993966    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:47:10.127951 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:10.127922    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:47:10.129042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:10.129005    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:47:10.132156 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:10.132138    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:47:10.133175 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:10.133149    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:47:31.021812 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:47:31.021770    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:47:56.099572 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.099490    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"]
Nov 01 08:47:56.103678 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.103657    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.155810 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.155782    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"]
Nov 01 08:47:56.201102 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.201056    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"premium-w3-54e01b-kserve-self-signed-certs\""
Nov 01 08:47:56.225713 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.225684    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tls-certs\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.225902 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.225730    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-dshm\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.225902 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.225818    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tmp-dir\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.225902 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.225853    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mnf4\" (UniqueName: \"kubernetes.io/projected/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-kube-api-access-8mnf4\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.226079 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.226000    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-home\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.226127 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.226090    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-model-cache\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.327419 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327383    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tls-certs\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.327624 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327429    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-dshm\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.327624 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327462    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tmp-dir\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.327624 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327484    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnf4\" (UniqueName: \"kubernetes.io/projected/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-kube-api-access-8mnf4\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.327624 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327544    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-home\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.327624 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327583    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-model-cache\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.327929 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327904    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tmp-dir\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.327983 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327943    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-home\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.328037 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.327993    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-model-cache\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.330003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.329977    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-dshm\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.330370 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.330349    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tls-certs\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.389273 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.389237    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mnf4\" (UniqueName: \"kubernetes.io/projected/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-kube-api-access-8mnf4\") pod \"premium-w3-54e01b-kserve-fdfb9dd8b-spgf2\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") " pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.417248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.417215    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:47:56.812395 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:56.812367    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"]
Nov 01 08:47:56.812935 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:47:56.812913    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddac89622_b6dc_47b0_9b5b_e723e7b4fb55.slice/crio-5796ceadf9da047ca20a8024b1a0caca7630ba442409981358c086dcc721a2f3 WatchSource:0}: Error finding container 5796ceadf9da047ca20a8024b1a0caca7630ba442409981358c086dcc721a2f3: Status 404 returned error can't find the container with id 5796ceadf9da047ca20a8024b1a0caca7630ba442409981358c086dcc721a2f3
Nov 01 08:47:57.263043 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:57.262999    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" event={"ID":"dac89622-b6dc-47b0-9b5b-e723e7b4fb55","Type":"ContainerStarted","Data":"0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43"}
Nov 01 08:47:57.263449 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:57.263067    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" event={"ID":"dac89622-b6dc-47b0-9b5b-e723e7b4fb55","Type":"ContainerStarted","Data":"5796ceadf9da047ca20a8024b1a0caca7630ba442409981358c086dcc721a2f3"}
Nov 01 08:47:57.375345 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:57.375295    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podStartSLOduration=1.375281928 podStartE2EDuration="1.375281928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:47:57.374487577 +0000 UTC m=+947.489360544" watchObservedRunningTime="2026-11-01 08:47:57.375281928 +0000 UTC m=+947.490154898"
Nov 01 08:47:57.417502 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:47:57.417474    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:48:01.023958 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:48:01.023917    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:48:06.418072 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:06.418015    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:48:06.431530 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:06.431500    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:48:07.335353 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:07.335323    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:48:25.445208 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.445170    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"]
Nov 01 08:48:25.449087 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.449063    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.518358 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.518315    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"]
Nov 01 08:48:25.533297 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.533262    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-home\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.533477 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.533332    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-model-cache\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.533477 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.533445    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a37272-11e6-4788-8f65-2f27b56dd9b2-tls-certs\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.533573 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.533484    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-dshm\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.533573 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.533563    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-tmp-dir\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.533654 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.533582    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbhn\" (UniqueName: \"kubernetes.io/projected/68a37272-11e6-4788-8f65-2f27b56dd9b2-kube-api-access-hjbhn\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.549861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.549822    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"openshift-service-ca.crt\""
Nov 01 08:48:25.549861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.549843    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"kube-root-ca.crt\""
Nov 01 08:48:25.550120 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.549906    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"default-dockercfg-l5p8k\""
Nov 01 08:48:25.550120 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.549970    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"premium-w1-dfdd13-kserve-self-signed-certs\""
Nov 01 08:48:25.634672 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.634633    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-model-cache\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.635010 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.634985    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a37272-11e6-4788-8f65-2f27b56dd9b2-tls-certs\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.635277 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.635183    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-dshm\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.635277 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.635192    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-model-cache\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.635417 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.635304    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-tmp-dir\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.635417 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.635337    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbhn\" (UniqueName: \"kubernetes.io/projected/68a37272-11e6-4788-8f65-2f27b56dd9b2-kube-api-access-hjbhn\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.635417 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.635369    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-home\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.635716 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.635697    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-home\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.635867 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.635845    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-tmp-dir\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.638334 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.638255    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-dshm\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.639141 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.639115    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a37272-11e6-4788-8f65-2f27b56dd9b2-tls-certs\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.699401 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.699319    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbhn\" (UniqueName: \"kubernetes.io/projected/68a37272-11e6-4788-8f65-2f27b56dd9b2-kube-api-access-hjbhn\") pod \"premium-w1-dfdd13-kserve-6655b66df9-wrsnk\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") " pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:25.761709 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:25.761675    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:26.139962 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:26.139925    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"]
Nov 01 08:48:26.140614 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:48:26.140578    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68a37272_11e6_4788_8f65_2f27b56dd9b2.slice/crio-aa7ac6a70b986ad3c437f26497b0945b72f7c08deffe2638ef64fcd19a5616a6 WatchSource:0}: Error finding container aa7ac6a70b986ad3c437f26497b0945b72f7c08deffe2638ef64fcd19a5616a6: Status 404 returned error can't find the container with id aa7ac6a70b986ad3c437f26497b0945b72f7c08deffe2638ef64fcd19a5616a6
Nov 01 08:48:26.438151 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:26.438063    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" event={"ID":"68a37272-11e6-4788-8f65-2f27b56dd9b2","Type":"ContainerStarted","Data":"7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59"}
Nov 01 08:48:26.438151 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:26.438104    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" event={"ID":"68a37272-11e6-4788-8f65-2f27b56dd9b2","Type":"ContainerStarted","Data":"aa7ac6a70b986ad3c437f26497b0945b72f7c08deffe2638ef64fcd19a5616a6"}
Nov 01 08:48:26.551770 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:26.551718    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podStartSLOduration=1.551703135 podStartE2EDuration="1.551703135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:48:26.549656487 +0000 UTC m=+976.664529452" watchObservedRunningTime="2026-11-01 08:48:26.551703135 +0000 UTC m=+976.666576102"
Nov 01 08:48:26.762071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:26.761972    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:31.021417 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:48:31.021380    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:48:35.762472 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:35.762435    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:35.776443 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:35.776411    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:36.508863 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:36.508824    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:48:42.954077 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:42.954038    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"]
Nov 01 08:48:42.958919 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:42.958900    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.041895 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.041855    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.042107 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.042015    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.042107 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.042068    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.042197 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.042177    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.042249 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.042206    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sfzk\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-kube-api-access-2sfzk\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.042292 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.042247    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.042334 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.042315    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.042387 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.042374    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.042461 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.042441    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.057076 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.057052    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b711ae-openshift-default-dockercfg-ms66n\""
Nov 01 08:48:43.061193 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.061167    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"]
Nov 01 08:48:43.143887 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.143847    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.143900    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.143970    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144007    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144080    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144167    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144196    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144273    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144305    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sfzk\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-kube-api-access-2sfzk\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144552    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144759 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144734    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144741    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144813    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.144958 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.144871    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.148231 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.148201    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.148404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.148387    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.202912 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.202876    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.203135 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.202929    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sfzk\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-kube-api-access-2sfzk\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.273468 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.273385    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:43.693303 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.693277    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"]
Nov 01 08:48:43.693966 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:48:43.693928    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd450b25e_ae73_4dac_9f39_9ee97becdd4a.slice/crio-5c423d4aebe1e724d925d96f989a836bda2f012db0343ff0cc7429a24b5c9a1d WatchSource:0}: Error finding container 5c423d4aebe1e724d925d96f989a836bda2f012db0343ff0cc7429a24b5c9a1d: Status 404 returned error can't find the container with id 5c423d4aebe1e724d925d96f989a836bda2f012db0343ff0cc7429a24b5c9a1d
Nov 01 08:48:43.696363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.696333    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:48:43.696423 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.696399    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:48:43.696463 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:43.696426    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:48:44.546765 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:44.546735    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" event={"ID":"d450b25e-ae73-4dac-9f39-9ee97becdd4a","Type":"ContainerStarted","Data":"e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef"}
Nov 01 08:48:44.547213 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:44.546783    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" event={"ID":"d450b25e-ae73-4dac-9f39-9ee97becdd4a","Type":"ContainerStarted","Data":"5c423d4aebe1e724d925d96f989a836bda2f012db0343ff0cc7429a24b5c9a1d"}
Nov 01 08:48:44.710236 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:44.710182    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" podStartSLOduration=2.710167347 podStartE2EDuration="2.710167347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:48:44.709670928 +0000 UTC m=+994.824543895" watchObservedRunningTime="2026-11-01 08:48:44.710167347 +0000 UTC m=+994.825040360"
Nov 01 08:48:45.282412 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:45.282377    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:45.285717 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:45.285686    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 08:48:45.285969 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:45.285940    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 08:48:46.284588 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:46.284543    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 08:48:46.285075 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:46.284612    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 08:48:47.275973 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:47.275493    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 08:48:47.275973 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:47.275559    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 08:48:48.276059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:48.275516    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 08:48:48.276059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:48.275596    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 08:48:49.275443 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:49.275406    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 08:48:49.275760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:49.275728    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 08:48:50.605161 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:50.605122    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:50.605698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:50.605672    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:48:50.606805 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:48:50.606787    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 08:49:01.023225 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:49:01.023184    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:49:23.037568 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.037481    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"]
Nov 01 08:49:23.043172 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.043147    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.094459 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.094413    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-751a73-openshift-default-dockercfg-qq2cf\""
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.108649    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.108711    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/798f2b4f-841f-457f-b97d-75e41c4f2749-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.108756    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.108800    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.108835    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn2wz\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-kube-api-access-dn2wz\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.108901    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.108946    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.109078    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.109152    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.144092 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.144064    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"]
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.212578    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.212643    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/798f2b4f-841f-457f-b97d-75e41c4f2749-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.212703    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.212734    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.212769    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn2wz\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-kube-api-access-dn2wz\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.212827    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.212880    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.212953    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.213041    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.213525    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.213788    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.214449    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/798f2b4f-841f-457f-b97d-75e41c4f2749-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.214734    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.232462    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.233067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.232896    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.235209 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.235191    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.316469 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.315476    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.316469 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.316246    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn2wz\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-kube-api-access-dn2wz\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.356918 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.356887    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:23.926257 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.917814    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"]
Nov 01 08:49:23.926257 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.923354    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:23.926257 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.923441    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:23.926257 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:23.923512    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:24.513245 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:24.513205    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-x8wfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:49:24.513874 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:24.513829    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:49:24.837816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:24.837729    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" event={"ID":"798f2b4f-841f-457f-b97d-75e41c4f2749","Type":"ContainerStarted","Data":"890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c"}
Nov 01 08:49:24.837816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:24.837799    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" event={"ID":"798f2b4f-841f-457f-b97d-75e41c4f2749","Type":"ContainerStarted","Data":"faf1e37090b2ca78b903869041549efb2f87910c907c2dc84faf06a1f41a8686"}
Nov 01 08:49:25.010055 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:25.009325    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" podStartSLOduration=3.009308375 podStartE2EDuration="3.009308375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:25.008716874 +0000 UTC m=+1035.123589867" watchObservedRunningTime="2026-11-01 08:49:25.009308375 +0000 UTC m=+1035.124181344"
Nov 01 08:49:25.357690 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:25.357659    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:25.364049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:25.363273    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 08:49:25.364049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:25.363326    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 08:49:26.358063 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:26.358003    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 08:49:26.358571 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:26.358081    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 08:49:27.358206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:27.358173    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 08:49:27.358583 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:27.358231    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 08:49:28.358004 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:28.357961    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 08:49:28.358260 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:28.358059    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 08:49:29.357209 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:29.357171    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 08:49:29.357442 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:29.357237    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 08:49:30.357678 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:30.357642    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 08:49:30.358184 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:30.357700    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 08:49:31.028083 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:49:31.028020    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:49:31.470995 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:31.470961    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:31.472313 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:31.472290    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:31.473534 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:31.473514    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:49:34.883191 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.883153    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"]
Nov 01 08:49:34.887868 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.887849    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:34.941715 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.941681    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"]
Nov 01 08:49:34.985279 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.985246    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"e2e-conflict-a-ef480be9-kserve-self-signed-certs\""
Nov 01 08:49:34.988716 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.988691    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-home\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:34.988846 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.988739    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-model-cache\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:34.988928 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.988844    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-dshm\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:34.988928 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.988893    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ae6050-a281-4bc8-be48-9802849540c0-tls-certs\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:34.989040 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.988952    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvwr\" (UniqueName: \"kubernetes.io/projected/99ae6050-a281-4bc8-be48-9802849540c0-kube-api-access-7zvwr\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:34.989102 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:34.989086    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-tmp-dir\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.089834 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.089791    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-tmp-dir\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.090073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.089873    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-home\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.090073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.089914    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-model-cache\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.090073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.089962    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-dshm\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.090073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.089990    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ae6050-a281-4bc8-be48-9802849540c0-tls-certs\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.090073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.090047    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvwr\" (UniqueName: \"kubernetes.io/projected/99ae6050-a281-4bc8-be48-9802849540c0-kube-api-access-7zvwr\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.090362 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.090310    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-tmp-dir\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.090595 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.090566    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-model-cache\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.090791 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.090769    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-home\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.093483 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.093449    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-dshm\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.094059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.094001    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ae6050-a281-4bc8-be48-9802849540c0-tls-certs\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.149367 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.149299    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvwr\" (UniqueName: \"kubernetes.io/projected/99ae6050-a281-4bc8-be48-9802849540c0-kube-api-access-7zvwr\") pod \"e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.201296 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.201232    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:35.613586 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.613559    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"]
Nov 01 08:49:35.932667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.932562    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" event={"ID":"99ae6050-a281-4bc8-be48-9802849540c0","Type":"ContainerStarted","Data":"53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229"}
Nov 01 08:49:35.932667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:35.932619    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" event={"ID":"99ae6050-a281-4bc8-be48-9802849540c0","Type":"ContainerStarted","Data":"5f153eda3f98fcd6d46557ecd7b55fb6c0d36267508c8a6b583c17208e90c43f"}
Nov 01 08:49:36.050840 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:36.050771    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" podStartSLOduration=2.050752898 podStartE2EDuration="2.050752898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:36.045228528 +0000 UTC m=+1046.160101498" watchObservedRunningTime="2026-11-01 08:49:36.050752898 +0000 UTC m=+1046.165625869"
Nov 01 08:49:36.202111 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:36.201994    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:39.511052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:39.510223    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-x8wfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:49:39.511052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:39.510307    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:49:40.861153 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.861117    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"]
Nov 01 08:49:40.865063 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.865017    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.917461 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.917435    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-54e01b-openshift-default-dockercfg-gln2n\""
Nov 01 08:49:40.957031 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.956999    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.957156 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.957077    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.957156 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.957100    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.957156 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.957123    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.957305 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.957179    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.957349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.957314    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.957388 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.957367    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.957434 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.957421    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.957492 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.957462    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tnhd\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-kube-api-access-2tnhd\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:40.969680 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:40.969654    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"]
Nov 01 08:49:41.058645 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.058612    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tnhd\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-kube-api-access-2tnhd\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.058816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.058685    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.058816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.058708    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.058816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.058725    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.058816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.058743    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059066 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.058897    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059066 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.058977    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059183 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.059087    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059183 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.059116    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059298 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.059219    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059298 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.059241    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059401 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.059318    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059457 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.059442    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.059872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.059847    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.061388 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.061368    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.061507 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.061489    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.117633 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.117571    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.117775 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.117710    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tnhd\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-kube-api-access-2tnhd\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.178406 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.178376    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:41.606443 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.606404    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"]
Nov 01 08:49:41.636706 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:49:41.636678    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda021d984_a9f1_4c70_b6b5_c4d924d27e99.slice/crio-93af4489d3b11532203533160773f51ebd6f0041e252171cbadc0ad5a6f81b2c WatchSource:0}: Error finding container 93af4489d3b11532203533160773f51ebd6f0041e252171cbadc0ad5a6f81b2c: Status 404 returned error can't find the container with id 93af4489d3b11532203533160773f51ebd6f0041e252171cbadc0ad5a6f81b2c
Nov 01 08:49:41.639311 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.639281    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:41.639450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.639371    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:41.639450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.639401    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:41.980559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.980461    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" event={"ID":"a021d984-a9f1-4c70-b6b5-c4d924d27e99","Type":"ContainerStarted","Data":"e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19"}
Nov 01 08:49:41.980559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:41.980517    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" event={"ID":"a021d984-a9f1-4c70-b6b5-c4d924d27e99","Type":"ContainerStarted","Data":"93af4489d3b11532203533160773f51ebd6f0041e252171cbadc0ad5a6f81b2c"}
Nov 01 08:49:42.098822 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:42.098768    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podStartSLOduration=2.09875283 podStartE2EDuration="2.09875283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:42.096175738 +0000 UTC m=+1052.211048716" watchObservedRunningTime="2026-11-01 08:49:42.09875283 +0000 UTC m=+1052.213625798"
Nov 01 08:49:42.183719 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:42.181390    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:42.184071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:42.184012    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:42.184210 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:42.184188    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 08:49:43.183052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:43.182382    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:43.183052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:43.182439    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 08:49:44.180049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:44.179457    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:44.180049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:44.179520    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 08:49:45.183049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:45.182463    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:45.183049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:45.182527    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 08:49:45.201773 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:45.201734    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:45.214178 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:45.214155    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:46.019929 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:46.019898    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:49:46.181567 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:46.181535    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:46.181844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:46.181817    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 08:49:47.183051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:47.181459    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:47.183051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:47.181541    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 08:49:48.178713 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:48.178671    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 08:49:48.178936 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:48.178747    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 08:49:49.183263 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:49.183233    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:49.183695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:49.183678    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:49.184322 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:49.184297    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 08:49:53.974673 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:53.974637    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"]
Nov 01 08:49:53.979925 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:53.979906    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149294    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnfc\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-kube-api-access-zdnfc\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149329 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149338    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149365    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149475    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149687 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149590    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149748 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149697    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149793 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149751    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149854 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149831    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.149969 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.149948    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.176159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.176128    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"]
Nov 01 08:49:54.250698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.250617    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.250857 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.250783    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.250857 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.250823    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.250970 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.250878    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.250970 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.250918    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251105 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.250989    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251105 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.251071    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251105 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.251086    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251269 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.251158    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnfc\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-kube-api-access-zdnfc\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251269 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.251212    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251269 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.251214    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251425 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.251339    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251603 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.251585    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.251670 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.251623    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.253608 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.253590    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.253660 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.253626    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.309971 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.309940    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnfc\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-kube-api-access-zdnfc\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.309971 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.309952    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-fspks\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:54.598656 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:54.598579    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:55.019556 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:49:55.019522    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a71f55_bb5e_4c2d_a7fd_fb4c4636a2b8.slice/crio-abf2fb1217da76aceb59db3f0c48d7ec99ae846cfc353a85c952c807b6dab953 WatchSource:0}: Error finding container abf2fb1217da76aceb59db3f0c48d7ec99ae846cfc353a85c952c807b6dab953: Status 404 returned error can't find the container with id abf2fb1217da76aceb59db3f0c48d7ec99ae846cfc353a85c952c807b6dab953
Nov 01 08:49:55.019896 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:55.019571    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"]
Nov 01 08:49:55.021998 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:55.021965    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:55.022114 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:55.022049    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:55.022114 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:55.022080    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:49:55.070367 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:55.070342    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" event={"ID":"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8","Type":"ContainerStarted","Data":"abf2fb1217da76aceb59db3f0c48d7ec99ae846cfc353a85c952c807b6dab953"}
Nov 01 08:49:56.089063 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:56.088991    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" event={"ID":"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8","Type":"ContainerStarted","Data":"6fa5fb15ac9114d5a8a5785714f18a43d0056f3d80792720add07405dba0baf7"}
Nov 01 08:49:56.216345 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:56.216292    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podStartSLOduration=3.216278171 podStartE2EDuration="3.216278171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:56.213675619 +0000 UTC m=+1066.328548588" watchObservedRunningTime="2026-11-01 08:49:56.216278171 +0000 UTC m=+1066.331151140"
Nov 01 08:49:56.599543 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:56.599509    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:49:56.601140 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:56.601112    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fspks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 08:49:56.601278 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:56.601176    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 08:49:57.599349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:57.599315    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fspks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 08:49:57.599743 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:57.599383    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 08:49:58.599404 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:58.599364    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fspks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 08:49:58.599810 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:58.599424    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 08:49:59.599815 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:59.599777    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fspks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 08:49:59.600260 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:49:59.599829    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 08:50:00.599607 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:00.599560    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fspks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 08:50:00.599814 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:00.599622    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 08:50:01.025259 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:50:01.025220    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:50:01.599586 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:01.599541    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fspks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 08:50:01.599760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:01.599612    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 08:50:02.600720 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:02.600566    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fspks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 08:50:02.601277 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:02.600692    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 08:50:04.431913 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:04.431875    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:50:04.433189 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:04.433165    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:50:04.434445 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:04.434425    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 08:50:31.125064 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:50:31.125002    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:50:47.769759 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.769724    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"]
Nov 01 08:50:47.774099 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.774082    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.824295 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.824264    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfdd13-openshift-default-dockercfg-jf87f\""
Nov 01 08:50:47.875552 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.875523    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"]
Nov 01 08:50:47.885923 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.885892    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.886116 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.885938    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.886116 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.885977    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.886227 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.886159    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djg4b\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-kube-api-access-djg4b\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.886277 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.886220    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.886347 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.886325    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.886398 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.886371    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.886441 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.886426    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.886475 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.886451    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987399 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987364    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987399 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987407    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987651 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987447    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987651 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987465    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987651 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987531    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987651 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987562    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987651 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987594    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987896 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987660    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djg4b\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-kube-api-access-djg4b\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.987896 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.987699    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.988131 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.988071    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.988131 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.988075    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.988284 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.988153    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.988420 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.988383    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.988632 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.988616    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.990323 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.990301    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:47.990745 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:47.990727    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:48.048054 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:48.047945    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:48.048247 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:48.048107    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djg4b\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-kube-api-access-djg4b\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:48.088246 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:48.088207    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:48.551355 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:50:48.551318    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d8333e_26a1_46ca_9ad4_2d6dc20d441b.slice/crio-67d9d53f57441d72970740c7f1a4d322e7acac2b4831f6094648462851ff9109 WatchSource:0}: Error finding container 67d9d53f57441d72970740c7f1a4d322e7acac2b4831f6094648462851ff9109: Status 404 returned error can't find the container with id 67d9d53f57441d72970740c7f1a4d322e7acac2b4831f6094648462851ff9109
Nov 01 08:50:48.552485 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:48.552459    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"]
Nov 01 08:50:48.553838 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:48.553801    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:50:48.553904 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:48.553883    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:50:48.553945 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:48.553922    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:50:49.505577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:49.505490    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" event={"ID":"01d8333e-26a1-46ca-9ad4-2d6dc20d441b","Type":"ContainerStarted","Data":"fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3"}
Nov 01 08:50:49.506169 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:49.506142    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" event={"ID":"01d8333e-26a1-46ca-9ad4-2d6dc20d441b","Type":"ContainerStarted","Data":"67d9d53f57441d72970740c7f1a4d322e7acac2b4831f6094648462851ff9109"}
Nov 01 08:50:50.088746 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:50.088619    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:50.091267 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:50.091237    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 08:50:50.091433 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:50.091295    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 08:50:51.089317 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:51.089263    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 08:50:51.089758 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:51.089371    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 08:50:52.089137 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:52.089102    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 08:50:52.089330 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:52.089162    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 08:50:53.097051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:53.095469    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 08:50:53.097051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:53.095555    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 08:50:54.090652 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:54.090612    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 08:50:54.091471 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:54.091438    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 08:50:55.089013 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:55.088975    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 08:50:55.089460 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:55.089078    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 08:50:56.093380 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:56.093351    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:56.093811 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:56.093783    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:56.094663 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:56.094642    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 08:50:56.268283 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:56.268226    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podStartSLOduration=9.268212887 podStartE2EDuration="9.268212887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:50:49.675418418 +0000 UTC m=+1119.790291390" watchObservedRunningTime="2026-11-01 08:50:56.268212887 +0000 UTC m=+1126.383085855"
Nov 01 08:50:59.002122 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.002088    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"]
Nov 01 08:50:59.006048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.006016    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131370    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131519    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131562    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131603    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8s4f\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-kube-api-access-b8s4f\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131690    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d44de48-7523-455d-8788-5ba07f9d6457-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131726    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d44de48-7523-455d-8788-5ba07f9d6457-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131765    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131794    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.133071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.131914    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.197806 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.197769    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"]
Nov 01 08:50:59.233547 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233516    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.233735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233559    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.233735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233576    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8s4f\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-kube-api-access-b8s4f\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.233735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233621    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d44de48-7523-455d-8788-5ba07f9d6457-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.233735 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233717    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d44de48-7523-455d-8788-5ba07f9d6457-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.233964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233767    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.233964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233805    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.233964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233907    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.233964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.233961    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.234274 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.234251    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.234478 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.234304    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.234478 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.234390    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.234478 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.234492    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.234802 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.234776    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d44de48-7523-455d-8788-5ba07f9d6457-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.236171 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.236148    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.237142 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.237121    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d44de48-7523-455d-8788-5ba07f9d6457-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.292672 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.292585    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.292837 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.292696    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8s4f\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-kube-api-access-b8s4f\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.320423 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.320382    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:50:59.750403 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.750375    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"]
Nov 01 08:50:59.751846 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:50:59.751821    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d44de48_7523_455d_8788_5ba07f9d6457.slice/crio-cfbfabf29e91d6bca591b63130b59d5de7e21c7bc85d376e7929f80b648f801a WatchSource:0}: Error finding container cfbfabf29e91d6bca591b63130b59d5de7e21c7bc85d376e7929f80b648f801a: Status 404 returned error can't find the container with id cfbfabf29e91d6bca591b63130b59d5de7e21c7bc85d376e7929f80b648f801a
Nov 01 08:50:59.754249 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.754215    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:50:59.754346 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.754291    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:50:59.754346 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:50:59.754323    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:51:00.593197 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:00.593152    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" event={"ID":"8d44de48-7523-455d-8788-5ba07f9d6457","Type":"ContainerStarted","Data":"9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a"}
Nov 01 08:51:00.593743 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:00.593218    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" event={"ID":"8d44de48-7523-455d-8788-5ba07f9d6457","Type":"ContainerStarted","Data":"cfbfabf29e91d6bca591b63130b59d5de7e21c7bc85d376e7929f80b648f801a"}
Nov 01 08:51:00.708998 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:00.708923    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" podStartSLOduration=2.708904098 podStartE2EDuration="2.708904098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:00.707775405 +0000 UTC m=+1130.822648376" watchObservedRunningTime="2026-11-01 08:51:00.708904098 +0000 UTC m=+1130.823777071"
Nov 01 08:51:01.032188 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:51:01.032154    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:51:01.321170 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:01.321080    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:51:01.323209 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:01.323183    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:51:01.323344 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:01.323246    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 08:51:02.321627 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:02.321597    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:51:02.322077 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:02.321650    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 08:51:03.321687 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:03.321647    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:51:03.322127 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:03.321720    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 08:51:04.321751 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:04.321719    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:51:04.322181 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:04.321773    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 08:51:05.321537 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:05.321493    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:51:05.321918 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:05.321887    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 08:51:06.321193 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:06.321155    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 08:51:06.321367 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:06.321228    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 08:51:07.664703 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:07.663324    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:51:07.664703 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:07.664613    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:51:07.667244 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:07.666079    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 08:51:16.880051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:16.879149    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:51:16.880051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:16.879233    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:51:31.023841 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:51:31.023801    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:51:37.444056 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.444008    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"]
Nov 01 08:51:37.448560 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.448537    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.498805 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.498774    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-343207-openshift-default-dockercfg-j8ktj\""
Nov 01 08:51:37.538325 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538282    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.538513 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538396    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.538513 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538481    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.538651 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538587    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.538710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538649    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.538710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538697    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.538809 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538722    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64spg\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-kube-api-access-64spg\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.538809 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538785    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.538907 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.538831    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.550308 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.550278    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"]
Nov 01 08:51:37.639519 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639461    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.639519 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639517    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.639788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639552    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.639788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639626    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.639788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639653    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.639788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639685    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.639788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639699    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64spg\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-kube-api-access-64spg\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.639788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639763    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.640225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.639806    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.640225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.640041    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.640225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.640132    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.640391 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.640226    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.640391 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.640297    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.640391 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.640369    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.642349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.642327    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.642557 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.642539    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.698959 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.698889    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.699127 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.699000    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64spg\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-kube-api-access-64spg\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:37.762292 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:37.762258    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:38.187995 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:38.187970    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"]
Nov 01 08:51:38.189282 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:51:38.189250    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4d6ff6_1afe_430a_9977_f8e5e9f223a2.slice/crio-ac92795422f5ceed3e23f3d5e0dd22d78f6f5c017a2ff3f2abb9511c1a266914 WatchSource:0}: Error finding container ac92795422f5ceed3e23f3d5e0dd22d78f6f5c017a2ff3f2abb9511c1a266914: Status 404 returned error can't find the container with id ac92795422f5ceed3e23f3d5e0dd22d78f6f5c017a2ff3f2abb9511c1a266914
Nov 01 08:51:38.191638 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:38.191607    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:51:38.191711 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:38.191672    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:51:38.191711 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:38.191702    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:51:38.934697 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:38.934662    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" event={"ID":"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2","Type":"ContainerStarted","Data":"8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8"}
Nov 01 08:51:38.935154 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:38.934727    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" event={"ID":"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2","Type":"ContainerStarted","Data":"ac92795422f5ceed3e23f3d5e0dd22d78f6f5c017a2ff3f2abb9511c1a266914"}
Nov 01 08:51:39.052556 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:39.052502    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podStartSLOduration=2.052489706 podStartE2EDuration="2.052489706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:39.050883429 +0000 UTC m=+1169.165756397" watchObservedRunningTime="2026-11-01 08:51:39.052489706 +0000 UTC m=+1169.167362673"
Nov 01 08:51:39.778406 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:39.778373    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:39.785274 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:39.785230    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:51:39.785460 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:39.785335    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 08:51:40.763497 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:40.763455    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:51:40.764123 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:40.764090    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 08:51:41.770049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.765164    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:51:41.770049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.765227    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 08:51:41.814118 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.814089    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"]
Nov 01 08:51:41.818391 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.818370    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.905639    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.905832    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.905871    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.905910    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.906018    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.906061    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.906120    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.906225    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.907641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.906290    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5m7r\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-kube-api-access-r5m7r\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:41.910872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:41.910844    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"]
Nov 01 08:51:42.006953 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.006915    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007194 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.006986    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007194 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007011    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007194 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007061    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007194 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007143    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007194 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007169    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007481 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007248    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007481 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007422    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007474    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007538    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007686 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007614    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007734 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007682    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.007734 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.007729    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5m7r\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-kube-api-access-r5m7r\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.008615 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.008592    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.009567 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.009537    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.009961 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.009945    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.087327 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.087257    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5m7r\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-kube-api-access-r5m7r\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.088661 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.088635    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.132897 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.132855    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:42.586956 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.586908    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"]
Nov 01 08:51:42.609551 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:51:42.609520    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fe99a6_e1a0_4dfb_9062_33d37cb8e677.slice/crio-e581723563efe60e18b5858f9cba981ca02691786f23f5bb5d6e80c46a639f8d WatchSource:0}: Error finding container e581723563efe60e18b5858f9cba981ca02691786f23f5bb5d6e80c46a639f8d: Status 404 returned error can't find the container with id e581723563efe60e18b5858f9cba981ca02691786f23f5bb5d6e80c46a639f8d
Nov 01 08:51:42.612169 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.612138    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:51:42.612265 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.612210    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:51:42.612265 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.612237    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:51:42.766969 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.766856    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:51:42.766969 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.766909    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 08:51:42.990206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.990165    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" event={"ID":"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677","Type":"ContainerStarted","Data":"1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407"}
Nov 01 08:51:42.990615 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:42.990231    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" event={"ID":"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677","Type":"ContainerStarted","Data":"e581723563efe60e18b5858f9cba981ca02691786f23f5bb5d6e80c46a639f8d"}
Nov 01 08:51:43.109314 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:43.109249    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podStartSLOduration=2.109226254 podStartE2EDuration="2.109226254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:43.105722452 +0000 UTC m=+1173.220595426" watchObservedRunningTime="2026-11-01 08:51:43.109226254 +0000 UTC m=+1173.224099223"
Nov 01 08:51:43.133332 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:43.133301    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:43.134900 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:43.134873    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:51:43.135019 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:43.134933    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 08:51:43.780052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:43.766445    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:51:43.780052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:43.766513    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 08:51:44.134624 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:44.134594    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:51:44.135109 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:44.135083    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 08:51:44.769049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:44.763438    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 08:51:44.769049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:44.763506    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 08:51:45.133650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:45.133609    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:51:45.133889 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:45.133663    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 08:51:45.898048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:45.897914    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:45.901073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:45.899603    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:45.902040 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:45.902007    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 08:51:46.133959 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:46.133925    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:51:46.134166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:46.133972    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 08:51:47.134110 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:47.134079    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:51:47.134518 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:47.134129    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 08:51:48.133698 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:48.133662    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:51:48.133866 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:48.133713    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 08:51:49.133783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:49.133740    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 08:51:49.134305 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:49.133816    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 08:51:50.541655 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:50.541626    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:50.542627 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:50.542603    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:51:50.543919 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:51:50.543722    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 08:52:00.456690 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.456655    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"]
Nov 01 08:52:00.460849 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.460822    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.526148 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.526116    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"]
Nov 01 08:52:00.559974 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.559944    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b711ae\"/\"distinct-2-w4-b711ae-kserve-self-signed-certs\""
Nov 01 08:52:00.569352 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.569324    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-model-cache\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.569527 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.569363    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29e67df-614a-4389-8610-8bfadbfa0a91-tls-certs\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.569527 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.569394    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-dshm\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.569599 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.569528    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-home\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.569599 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.569564    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flm8q\" (UniqueName: \"kubernetes.io/projected/b29e67df-614a-4389-8610-8bfadbfa0a91-kube-api-access-flm8q\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.569680 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.569596    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-tmp-dir\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.671250 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.671201    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-dshm\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.671537 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.671319    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-home\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.671537 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.671367    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flm8q\" (UniqueName: \"kubernetes.io/projected/b29e67df-614a-4389-8610-8bfadbfa0a91-kube-api-access-flm8q\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.671537 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.671410    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-tmp-dir\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.671537 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.671529    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-model-cache\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.671809 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.671565    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29e67df-614a-4389-8610-8bfadbfa0a91-tls-certs\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.672535 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.672502    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-home\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.672654 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.672512    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-tmp-dir\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.672774 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.672756    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-model-cache\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.674041 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.674000    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-dshm\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.674851 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.674831    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29e67df-614a-4389-8610-8bfadbfa0a91-tls-certs\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.731519 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.731443    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flm8q\" (UniqueName: \"kubernetes.io/projected/b29e67df-614a-4389-8610-8bfadbfa0a91-kube-api-access-flm8q\") pod \"distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") " pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:00.774203 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:00.774164    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:01.024071 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:52:01.023978    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:52:01.210773 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:01.210740    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"]
Nov 01 08:52:01.211424 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:52:01.211395    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb29e67df_614a_4389_8610_8bfadbfa0a91.slice/crio-558e8693601a3c16ccded6e8bea48a154e2bfdecf5d840c736df20690ffbf1a7 WatchSource:0}: Error finding container 558e8693601a3c16ccded6e8bea48a154e2bfdecf5d840c736df20690ffbf1a7: Status 404 returned error can't find the container with id 558e8693601a3c16ccded6e8bea48a154e2bfdecf5d840c736df20690ffbf1a7
Nov 01 08:52:02.156960 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:02.156924    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" event={"ID":"b29e67df-614a-4389-8610-8bfadbfa0a91","Type":"ContainerStarted","Data":"237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551"}
Nov 01 08:52:02.157426 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:02.156970    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" event={"ID":"b29e67df-614a-4389-8610-8bfadbfa0a91","Type":"ContainerStarted","Data":"558e8693601a3c16ccded6e8bea48a154e2bfdecf5d840c736df20690ffbf1a7"}
Nov 01 08:52:02.272659 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:02.272595    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podStartSLOduration=2.272576606 podStartE2EDuration="2.272576606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:02.268963388 +0000 UTC m=+1192.383836354" watchObservedRunningTime="2026-11-01 08:52:02.272576606 +0000 UTC m=+1192.387449574"
Nov 01 08:52:02.775467 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:02.775433    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:10.174076 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:10.174019    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:52:10.175432 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:10.175411    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:52:10.178413 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:10.178393    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:52:10.179354 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:10.179337    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:52:10.775105 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:10.775016    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:10.793884 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:10.793855    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:11.240804 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:11.240775    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:18.483789 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.483754    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"]
Nov 01 08:52:18.488894 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.488875    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507352 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507324    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507466 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507360    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjdt\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-kube-api-access-xnjdt\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507466 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507409    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924263da-31d9-47f5-8b0e-b748a711033c-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507556 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507505    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507598 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507559    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507679 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507665    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507718 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507697    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507763 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507731    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924263da-31d9-47f5-8b0e-b748a711033c-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.507872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.507853    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608495 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608459    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608671 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608537    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608671 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608557    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjdt\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-kube-api-access-xnjdt\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608671 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608600    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924263da-31d9-47f5-8b0e-b748a711033c-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608850 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608719    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608850 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608770    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608974 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608847    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608974 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608883    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608974 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608914    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.608974 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608915    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.609211 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.608976    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924263da-31d9-47f5-8b0e-b748a711033c-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.609211 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.609110    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.609314 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.609217    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.609931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.609908    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924263da-31d9-47f5-8b0e-b748a711033c-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.611346 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.611323    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.611663 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.611621    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924263da-31d9-47f5-8b0e-b748a711033c-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.668290 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.668255    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.668464 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.668400    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjdt\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-kube-api-access-xnjdt\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:18.693752 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.693723    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"]
Nov 01 08:52:18.802400 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:18.802320    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:19.223424 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:19.223398    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"]
Nov 01 08:52:19.224172 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:52:19.224146    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924263da_31d9_47f5_8b0e_b748a711033c.slice/crio-37907cd6d416af45dc89e5afcad2fef9d49434286241341f8e2c8f9e8d6ab9a5 WatchSource:0}: Error finding container 37907cd6d416af45dc89e5afcad2fef9d49434286241341f8e2c8f9e8d6ab9a5: Status 404 returned error can't find the container with id 37907cd6d416af45dc89e5afcad2fef9d49434286241341f8e2c8f9e8d6ab9a5
Nov 01 08:52:19.226858 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:19.226826    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:19.226938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:19.226894    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:19.226938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:19.226927    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:19.283621 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:19.283588    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" event={"ID":"924263da-31d9-47f5-8b0e-b748a711033c","Type":"ContainerStarted","Data":"37907cd6d416af45dc89e5afcad2fef9d49434286241341f8e2c8f9e8d6ab9a5"}
Nov 01 08:52:20.298890 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.298845    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" event={"ID":"924263da-31d9-47f5-8b0e-b748a711033c","Type":"ContainerStarted","Data":"38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f"}
Nov 01 08:52:20.802897 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.802862    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:20.804606 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.804574    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 08:52:20.804730 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.804638    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 08:52:20.872048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.860056    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podStartSLOduration=2.860013501 podStartE2EDuration="2.860013501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:20.464319926 +0000 UTC m=+1210.579192896" watchObservedRunningTime="2026-11-01 08:52:20.860013501 +0000 UTC m=+1210.974886472"
Nov 01 08:52:20.874514 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.874483    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"]
Nov 01 08:52:20.893417 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.893388    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:20.921444 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.921412    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"]
Nov 01 08:52:20.938402 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.938369    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwkv\" (UniqueName: \"kubernetes.io/projected/0d12efcf-78f3-424b-ab14-e0f8f756e930-kube-api-access-4dwkv\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:20.938582 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.938499    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-dshm\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:20.938582 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.938536    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-home\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:20.938709 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.938637    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d12efcf-78f3-424b-ab14-e0f8f756e930-tls-certs\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:20.938791 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.938774    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-model-cache\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:20.938856 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.938809    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-tmp-dir\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:20.944482 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.944458    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"default-dockercfg-f8cwt\""
Nov 01 08:52:20.944482 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.944473    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"openshift-service-ca.crt\""
Nov 01 08:52:20.944669 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.944549    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"kube-root-ca.crt\""
Nov 01 08:52:20.944669 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:20.944554    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-343207\"/\"distinct-2-w2-343207-kserve-self-signed-certs\""
Nov 01 08:52:21.040288 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.040252    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-model-cache\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.040497 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.040304    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-tmp-dir\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.040497 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.040367    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwkv\" (UniqueName: \"kubernetes.io/projected/0d12efcf-78f3-424b-ab14-e0f8f756e930-kube-api-access-4dwkv\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.040497 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.040489    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-dshm\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.040674 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.040526    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-home\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.040674 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.040559    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d12efcf-78f3-424b-ab14-e0f8f756e930-tls-certs\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.040962 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.040936    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-model-cache\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.041242 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.041219    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-tmp-dir\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.041352 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.041108    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-home\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.043827 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.043801    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-dshm\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.044986 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.044964    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d12efcf-78f3-424b-ab14-e0f8f756e930-tls-certs\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.101107 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.100137    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwkv\" (UniqueName: \"kubernetes.io/projected/0d12efcf-78f3-424b-ab14-e0f8f756e930-kube-api-access-4dwkv\") pod \"distinct-2-w2-343207-kserve-cbd59b487-thv8j\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") " pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.212622 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.212583    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:21.686844 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:52:21.686811    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d12efcf_78f3_424b_ab14_e0f8f756e930.slice/crio-b689c871146f304f621a455ecbe0918e0e5f81764c3156d69e9f619380540ea7 WatchSource:0}: Error finding container b689c871146f304f621a455ecbe0918e0e5f81764c3156d69e9f619380540ea7: Status 404 returned error can't find the container with id b689c871146f304f621a455ecbe0918e0e5f81764c3156d69e9f619380540ea7
Nov 01 08:52:21.687251 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.686981    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"]
Nov 01 08:52:21.803016 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.802986    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 08:52:21.803179 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:21.803063    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 08:52:22.345957 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:22.345924    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" event={"ID":"0d12efcf-78f3-424b-ab14-e0f8f756e930","Type":"ContainerStarted","Data":"161b086b599982286e140c30bc87e8f3740a0b5621d349ae49abca1d7d65138f"}
Nov 01 08:52:22.346211 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:22.345970    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" event={"ID":"0d12efcf-78f3-424b-ab14-e0f8f756e930","Type":"ContainerStarted","Data":"b689c871146f304f621a455ecbe0918e0e5f81764c3156d69e9f619380540ea7"}
Nov 01 08:52:22.803541 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:22.803504    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 08:52:22.804066 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:22.803566    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 08:52:23.213225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:23.213192    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:23.803039 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:23.802993    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 08:52:23.803224 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:23.803080    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 08:52:24.803519 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:24.803478    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 08:52:24.803945 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:24.803531    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 08:52:25.802724 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:25.802688    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 08:52:25.802934 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:25.802755    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 08:52:27.157249 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:27.157215    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:27.157751 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:27.157677    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:27.159486 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:27.159463    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 08:52:27.322134 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:27.322070    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podStartSLOduration=7.322056302 podStartE2EDuration="7.322056302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:22.49033047 +0000 UTC m=+1212.605203433" watchObservedRunningTime="2026-11-01 08:52:27.322056302 +0000 UTC m=+1217.436929263"
Nov 01 08:52:29.262617 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.262584    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"]
Nov 01 08:52:29.267373 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.267348    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.371449 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.371400    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"]
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400340    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400410    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172f6137-4e24-4ad6-b76f-e9c236db3ece-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400561    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400635    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400700    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400831    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlhl\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-kube-api-access-shlhl\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400909    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400943    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.401301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.400999    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502578 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502536    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502591    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172f6137-4e24-4ad6-b76f-e9c236db3ece-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502630    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502678    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502783 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502722    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502997 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502814    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlhl\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-kube-api-access-shlhl\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502997 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502861    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502997 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502890    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502997 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502926    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.502997 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.502952    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.503398 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.503372    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.503907 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.503873    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.504075 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.504054    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172f6137-4e24-4ad6-b76f-e9c236db3ece-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.504185 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.504165    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.506391 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.506368    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.506603 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.506585    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.564658 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.564578    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.564940 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.564920    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlhl\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-kube-api-access-shlhl\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:29.581801 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:29.581773    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:30.008207 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:30.008178    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"]
Nov 01 08:52:30.009734 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:52:30.009705    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172f6137_4e24_4ad6_b76f_e9c236db3ece.slice/crio-8e542a14ce5e77d2e92a99b1201b18fc1c8a0341a2fc3b6c4ddf97660d8afeeb WatchSource:0}: Error finding container 8e542a14ce5e77d2e92a99b1201b18fc1c8a0341a2fc3b6c4ddf97660d8afeeb: Status 404 returned error can't find the container with id 8e542a14ce5e77d2e92a99b1201b18fc1c8a0341a2fc3b6c4ddf97660d8afeeb
Nov 01 08:52:30.012083 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:30.012013    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:30.012207 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:30.012131    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:30.012207 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:30.012171    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:30.413205 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:30.413161    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" event={"ID":"172f6137-4e24-4ad6-b76f-e9c236db3ece","Type":"ContainerStarted","Data":"1561835b74635c859f59418d74d6220aa17bd5fe400f89427a69fe53b69def30"}
Nov 01 08:52:30.413682 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:30.413223    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" event={"ID":"172f6137-4e24-4ad6-b76f-e9c236db3ece","Type":"ContainerStarted","Data":"8e542a14ce5e77d2e92a99b1201b18fc1c8a0341a2fc3b6c4ddf97660d8afeeb"}
Nov 01 08:52:30.531096 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:30.531046    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podStartSLOduration=1.531017622 podStartE2EDuration="1.531017622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:30.528625521 +0000 UTC m=+1220.643498489" watchObservedRunningTime="2026-11-01 08:52:30.531017622 +0000 UTC m=+1220.645890590"
Nov 01 08:52:30.582272 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:30.582235    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:31.025993 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:52:31.025955    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:52:31.213174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:31.213128    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:31.229619 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:31.229593    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:31.430923 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:31.430893    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:31.591078 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:31.588790    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:52:31.591078 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:31.588862    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 08:52:32.590099 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:32.586054    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:52:32.590099 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:32.586121    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 08:52:33.590265 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.589607    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:52:33.590265 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.589674    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 08:52:33.603742 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.603698    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"]
Nov 01 08:52:33.609547 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.609518    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.711613 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.711582    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"]
Nov 01 08:52:33.756310 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.756278    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.756579 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.756566    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.756696 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.756687    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.756793 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.756784    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.756907 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.756897    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.756990 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.756980    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtwh\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-kube-api-access-phtwh\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.757116 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.757104    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b095b7-1bea-4711-be17-a9ed5d7907f1-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.757225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.757216    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.757333 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.757324    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.858764 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.858723    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.858963 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.858787    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.858963 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.858833    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.858963 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.858874    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.858963 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.858905    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.858963 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.858928    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtwh\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-kube-api-access-phtwh\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.858963 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.858951    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b095b7-1bea-4711-be17-a9ed5d7907f1-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.859317 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.859003    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.862160 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.862129    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.862915 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.862889    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.864718 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.864694    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b095b7-1bea-4711-be17-a9ed5d7907f1-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.875439 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.873637    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.875621 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.873868    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.875745 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.874115    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.903668 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.903628    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.906602 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.906573    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.922247 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.922225    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.932375 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.932322    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtwh\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-kube-api-access-phtwh\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:33.942267 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:33.942057    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:34.418186 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:52:34.418155    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b095b7_1bea_4711_be17_a9ed5d7907f1.slice/crio-291f3ad04b73b85d60f73678848cb333964510b81d3023f636bfcc2311bf2f23 WatchSource:0}: Error finding container 291f3ad04b73b85d60f73678848cb333964510b81d3023f636bfcc2311bf2f23: Status 404 returned error can't find the container with id 291f3ad04b73b85d60f73678848cb333964510b81d3023f636bfcc2311bf2f23
Nov 01 08:52:34.418467 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:34.418439    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"]
Nov 01 08:52:34.421225 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:34.421188    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:34.421311 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:34.421277    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:34.421356 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:34.421326    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:34.455533 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:34.455493    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" event={"ID":"05b095b7-1bea-4711-be17-a9ed5d7907f1","Type":"ContainerStarted","Data":"291f3ad04b73b85d60f73678848cb333964510b81d3023f636bfcc2311bf2f23"}
Nov 01 08:52:34.597045 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:34.588459    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:52:34.597045 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:34.588519    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 08:52:35.477435 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:35.477397    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" event={"ID":"05b095b7-1bea-4711-be17-a9ed5d7907f1","Type":"ContainerStarted","Data":"c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd"}
Nov 01 08:52:35.587053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:35.583467    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:52:35.587053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:35.583528    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 08:52:35.720395 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:35.720332    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podStartSLOduration=2.72031081 podStartE2EDuration="2.72031081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:35.655459397 +0000 UTC m=+1225.770332366" watchObservedRunningTime="2026-11-01 08:52:35.72031081 +0000 UTC m=+1225.835183778"
Nov 01 08:52:35.943420 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:35.943391    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:35.944500 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:35.944478    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:52:35.944570 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:35.944529    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 08:52:36.585470 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:36.585429    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:52:36.585725 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:36.585698    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 08:52:36.943435 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:36.943401    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:52:36.943861 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:36.943457    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 08:52:37.587051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:37.582420    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 08:52:37.587051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:37.582483    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 08:52:37.943462 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:37.943419    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:52:37.943862 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:37.943487    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 08:52:38.595797 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:38.595768    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:38.596848 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:38.596812    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:38.597727 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:38.597708    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 08:52:38.943680 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:38.943652    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:52:38.944350 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:38.944325    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 08:52:39.942890 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:39.942849    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:52:39.943094 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:39.942917    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 08:52:40.943147 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:40.943114    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:52:40.943553 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:40.943168    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 08:52:41.943088 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:41.943009    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 08:52:41.943363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:41.943119    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 08:52:42.947226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:42.947195    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:42.947683 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:42.947498    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:42.948141 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:42.948125    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 08:52:45.022819 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:45.022788    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"]
Nov 01 08:52:53.040927 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.040888    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"]
Nov 01 08:52:53.047471 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.047443    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.190675 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.190631    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.190976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.190952    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.191088 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.191019    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.191205 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.191187    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.191308 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.191284    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mlh\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-kube-api-access-n9mlh\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.191549 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.191524    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.191753 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.191722    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.191842 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.191766    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.191842 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.191825    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.265117 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.265075    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"]
Nov 01 08:52:53.293566 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.293466    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.293846 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.293822    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.293976 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.293960    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.294206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.294187    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.294322 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.294290    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.294455 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.294435    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.294582 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.294565    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.294691 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.294677    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.294820 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.294805    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mlh\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-kube-api-access-n9mlh\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.296015 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.295003    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.296015 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.295137    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.296015 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.295280    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.296015 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.295517    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.296015 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.295969    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.299656 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.299591    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.304803 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.304761    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.355171 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.355130    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.355354 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.355298    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mlh\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-kube-api-access-n9mlh\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:53.371101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:53.371069    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:54.280054 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:54.264347    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"]
Nov 01 08:52:54.308050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:54.304907    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:54.308050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:54.305006    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:54.308050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:54.305073    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:52:54.714931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:54.714890    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" event={"ID":"02f41188-8e9e-4007-a9d5-43c775e2e3a9","Type":"ContainerStarted","Data":"3abf2130d3e7dd22f56ae27cbaa36c9489f4b09aa11af152c32c2ab67baba38e"}
Nov 01 08:52:54.988052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:54.987324    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.018061 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.017136    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" podUID="99ae6050-a281-4bc8-be48-9802849540c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.349222 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.349120    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:52:55.447056 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.440264    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:52:55.516054 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.515329    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.755052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.753997    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" event={"ID":"02f41188-8e9e-4007-a9d5-43c775e2e3a9","Type":"ContainerStarted","Data":"9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57"}
Nov 01 08:52:55.780053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.766157    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.937052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.936193    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podStartSLOduration=3.936171808 podStartE2EDuration="3.936171808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:55.931635087 +0000 UTC m=+1246.046508052" watchObservedRunningTime="2026-11-01 08:52:55.936171808 +0000 UTC m=+1246.051044777"
Nov 01 08:52:55.995051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.992303    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:55.995051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:55.992995    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:52:56.029143 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.022121    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" podUID="99ae6050-a281-4bc8-be48-9802849540c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:56.029143 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.022552    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:52:56.247048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.246776    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:52:56.391276 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.391236    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:52:56.613051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.582266    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:52:56.613375 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.586595    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:56.639086 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.625279    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 08:52:56.713164 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.692648    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:52:56.713491 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.713469    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:56.713666 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.713653    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:52:56.713807 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.713796    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:52:56.777062 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.776690    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:52:56.784074 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.779605    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:56.806302 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:56.806256    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.247328 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.247278    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.358076 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.284975    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:57.395599 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.395131    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:52:57.395599 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.395195    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:52:57.434302 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.434122    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:52:57.434302 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.434220    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.621111 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.609307    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.621340 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.621250    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.638101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.633106    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.802889 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.802845    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.804262 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.803159    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.844171 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.844119    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:57.844934 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:57.844908    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:52:58.051101 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.033580    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" podUID="99ae6050-a281-4bc8-be48-9802849540c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.226205 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.226164    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:52:58.237053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.230148    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.280743 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.280682    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:52:58.425210 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.425143    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:52:58.430050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.426732    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:52:58.452365 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.452318    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:52:58.636042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.634825    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.646725 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.646685    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 08:52:58.814154 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.804987    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:58.814154 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.805409    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 01 08:52:58.814154 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.805744    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:52:58.814154 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.805802    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:52:58.846064 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:58.845433    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.009051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.005488    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:52:59.009051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.005565    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.046051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.041189    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" podUID="99ae6050-a281-4bc8-be48-9802849540c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.046051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.041639    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:52:59.285192 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.285146    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.482113 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.413439    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:52:59.499748 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.499709    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:52:59.501912 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.447243    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:52:59.509397 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.500943    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:52:59.521193 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.521055    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Nov 01 08:52:59.521446 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.521184    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:52:59.521601 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.521589    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:52:59.530729 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.530689    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:52:59.664048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.651137    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:52:59.725045 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.715208    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:52:59.734203 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.727008    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:52:59.739464 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.739431    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:52:59.747792 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.747766    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:52:59.755366 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:52:59.755340    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:53:00.046949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.044882    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" podUID="99ae6050-a281-4bc8-be48-9802849540c0" containerName="main" containerID="cri-o://53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229" gracePeriod=45
Nov 01 08:53:00.388596 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.388558    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:53:00.388931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.388880    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:53:00.456158 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.456132    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:53:00.525643 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.525607    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-tmp-dir\") pod \"99ae6050-a281-4bc8-be48-9802849540c0\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") "
Nov 01 08:53:00.526174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.525716    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ae6050-a281-4bc8-be48-9802849540c0-kube-api-access-7zvwr\" (UniqueName: \"kubernetes.io/projected/99ae6050-a281-4bc8-be48-9802849540c0-kube-api-access-7zvwr\") pod \"99ae6050-a281-4bc8-be48-9802849540c0\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") "
Nov 01 08:53:00.526174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.525795    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99ae6050-a281-4bc8-be48-9802849540c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/99ae6050-a281-4bc8-be48-9802849540c0-tls-certs\") pod \"99ae6050-a281-4bc8-be48-9802849540c0\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") "
Nov 01 08:53:00.526174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.525840    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-home\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-home\") pod \"99ae6050-a281-4bc8-be48-9802849540c0\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") "
Nov 01 08:53:00.526174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.525861    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-tmp-dir" pod "99ae6050-a281-4bc8-be48-9802849540c0" (UID: "99ae6050-a281-4bc8-be48-9802849540c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:53:00.526174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.525883    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-dshm\") pod \"99ae6050-a281-4bc8-be48-9802849540c0\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") "
Nov 01 08:53:00.526174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.525947    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-model-cache\") pod \"99ae6050-a281-4bc8-be48-9802849540c0\" (UID: \"99ae6050-a281-4bc8-be48-9802849540c0\") "
Nov 01 08:53:00.526483 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.526275    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-home" pod "99ae6050-a281-4bc8-be48-9802849540c0" (UID: "99ae6050-a281-4bc8-be48-9802849540c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:53:00.526588 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.526565    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-model-cache" pod "99ae6050-a281-4bc8-be48-9802849540c0" (UID: "99ae6050-a281-4bc8-be48-9802849540c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:53:00.526654 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.526624    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:00.526654 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.526644    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:00.528407 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.528386    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ae6050-a281-4bc8-be48-9802849540c0-tls-certs" pod "99ae6050-a281-4bc8-be48-9802849540c0" (UID: "99ae6050-a281-4bc8-be48-9802849540c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:53:00.528755 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.528735    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ae6050-a281-4bc8-be48-9802849540c0-kube-api-access-7zvwr" pod "99ae6050-a281-4bc8-be48-9802849540c0" (UID: "99ae6050-a281-4bc8-be48-9802849540c0"). InnerVolumeSpecName "kube-api-access-7zvwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:53:00.528925 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.528908    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-dshm" pod "99ae6050-a281-4bc8-be48-9802849540c0" (UID: "99ae6050-a281-4bc8-be48-9802849540c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:53:00.627829 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.627792    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvwr\" (UniqueName: \"kubernetes.io/projected/99ae6050-a281-4bc8-be48-9802849540c0-kube-api-access-7zvwr\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:00.628002 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.627834    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99ae6050-a281-4bc8-be48-9802849540c0-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:00.628002 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.627850    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:00.628002 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:00.627865    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99ae6050-a281-4bc8-be48-9802849540c0-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:53:01.033078 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:53:01.032978    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:53:01.127472 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.127437    4931 generic.go:356] "Generic (PLEG): container finished" podID="99ae6050-a281-4bc8-be48-9802849540c0" containerID="53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229" exitCode=0
Nov 01 08:53:01.127674 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.127491    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" event={"ID":"99ae6050-a281-4bc8-be48-9802849540c0","Type":"ContainerDied","Data":"53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229"}
Nov 01 08:53:01.127674 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.127518    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979" event={"ID":"99ae6050-a281-4bc8-be48-9802849540c0","Type":"ContainerDied","Data":"5f153eda3f98fcd6d46557ecd7b55fb6c0d36267508c8a6b583c17208e90c43f"}
Nov 01 08:53:01.127674 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.127534    4931 scope.go:122] "RemoveContainer" containerID="53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229"
Nov 01 08:53:01.127674 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.127491    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"
Nov 01 08:53:01.138433 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.138408    4931 scope.go:122] "RemoveContainer" containerID="53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229"
Nov 01 08:53:01.138744 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:53:01.138726    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229\": container with ID starting with 53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229 not found: ID does not exist" containerID="53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229"
Nov 01 08:53:01.138799 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.138755    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229"} err="failed to get container status \"53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229\": rpc error: code = NotFound desc = could not find container \"53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229\": container with ID starting with 53aac04dabf71e575230e0980062abb8b3987495ca53565cd81be4c8aa1ce229 not found: ID does not exist"
Nov 01 08:53:01.296089 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.295968    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"]
Nov 01 08:53:01.300131 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.300103    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-a-ef480be9-kserve-76f4b66846-4n979"]
Nov 01 08:53:01.390055 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.382451    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:53:01.390055 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:01.382517    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:53:02.050869 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:02.050832    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ae6050-a281-4bc8-be48-9802849540c0" path="/var/lib/kubelet/pods/99ae6050-a281-4bc8-be48-9802849540c0/volumes"
Nov 01 08:53:02.382940 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:02.382903    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:53:02.383250 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:02.383223    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:53:03.376476 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:03.376435    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:53:03.377104 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:03.377069    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:53:04.371537 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:04.371506    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 08:53:04.371692 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:04.371556    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 08:53:05.522788 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:05.522753    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:53:05.524226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:05.524195    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:53:05.524839 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:05.524807    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 08:53:12.506483 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.506447    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"]
Nov 01 08:53:12.507060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.507011    4931 cpu_manager.go:414] "containerMap: removing container" podUID="99ae6050-a281-4bc8-be48-9802849540c0" containerName="main"
Nov 01 08:53:12.507060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.507040    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ae6050-a281-4bc8-be48-9802849540c0" containerName="main"
Nov 01 08:53:12.507183 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.507137    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ae6050-a281-4bc8-be48-9802849540c0" containerName="main"
Nov 01 08:53:12.512307 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.512282    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.595363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.595310    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.595585 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.595464    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.595695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.595601    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.595695 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.595654    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecf2618-f684-49a2-adaf-6c5f84550877-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.595919 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.595778    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-kube-api-access-fkkgp\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.595919 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.595906    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecf2618-f684-49a2-adaf-6c5f84550877-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.596059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.595971    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.596113 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.596070    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.596113 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.596092    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.613590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.613558    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"]
Nov 01 08:53:12.696996 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.696961    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.696996 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.696998    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.697249 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697070    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.697249 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697134    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.697249 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697236    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.697409 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697264    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecf2618-f684-49a2-adaf-6c5f84550877-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.697409 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697291    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-kube-api-access-fkkgp\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.697409 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697327    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecf2618-f684-49a2-adaf-6c5f84550877-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.697409 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697362    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.697621 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697588    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.698012 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.697979    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.698170 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.698108    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.698170 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.698118    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecf2618-f684-49a2-adaf-6c5f84550877-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.698306 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.698279    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.699686 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.699668    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.700307 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.700290    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecf2618-f684-49a2-adaf-6c5f84550877-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.756609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.756541    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-kube-api-access-fkkgp\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.756609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.756542    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:12.826594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:12.826560    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:13.254494 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:13.254468    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"]
Nov 01 08:53:13.254676 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:53:13.254617    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecf2618_f684_49a2_adaf_6c5f84550877.slice/crio-8a6b9b07bd1e81e68b53d7293eb532124053b816472b47753b2e839c95bc6ebc WatchSource:0}: Error finding container 8a6b9b07bd1e81e68b53d7293eb532124053b816472b47753b2e839c95bc6ebc: Status 404 returned error can't find the container with id 8a6b9b07bd1e81e68b53d7293eb532124053b816472b47753b2e839c95bc6ebc
Nov 01 08:53:13.257208 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:13.257177    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:53:13.257294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:13.257244    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:53:13.257294 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:13.257273    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:53:14.221702 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:14.221671    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" event={"ID":"cecf2618-f684-49a2-adaf-6c5f84550877","Type":"ContainerStarted","Data":"303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156"}
Nov 01 08:53:14.222141 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:14.221723    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" event={"ID":"cecf2618-f684-49a2-adaf-6c5f84550877","Type":"ContainerStarted","Data":"8a6b9b07bd1e81e68b53d7293eb532124053b816472b47753b2e839c95bc6ebc"}
Nov 01 08:53:14.436128 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:14.436046    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podStartSLOduration=2.436012232 podStartE2EDuration="2.436012232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:14.388286945 +0000 UTC m=+1264.503159915" watchObservedRunningTime="2026-11-01 08:53:14.436012232 +0000 UTC m=+1264.550885200"
Nov 01 08:53:14.835580 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:14.835495    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:14.845649 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:14.845615    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:53:14.845924 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:14.845901    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 08:53:15.839052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:15.833943    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:53:15.839052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:15.834017    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 08:53:16.744177 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:16.744122    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:16.756158 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:16.756111    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:53:16.834506 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:16.834470    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:53:16.834766 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:16.834741    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 08:53:17.537728 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.537682    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:17.559085 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.558157    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:17.728207 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.728164    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:53:17.728577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.728549    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:17.750559 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.750505    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:17.751362 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.751336    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:53:17.767990 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.767948    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:53:17.769802 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.768914    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:53:17.828521 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.828426    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:53:17.828804 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:17.828772    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 08:53:18.625067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.615156    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.625067 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.615967    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:53:18.636304 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.631155    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.636304 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.633979    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:53:18.641834 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.641790    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.730051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.729425    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.734098 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.730780    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:53:18.734098 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.731911    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:53:18.734098 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.731965    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:53:18.778440 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.777222    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:53:18.779641 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.778686    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:18.856741 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.856707    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:53:18.857051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:18.857003    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 08:53:19.311085 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.306182    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:19.544059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.541714    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:53:19.553450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.553418    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:53:19.592014 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.591928    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:53:19.650174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.649981    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:53:19.655829 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.655808    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:53:19.672668 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.659358    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:53:19.693576 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.693528    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:19.694924 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.694859    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:53:19.859780 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.859731    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:53:19.863166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.863125    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 08:53:19.864710 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:19.859880    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:53:20.827431 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:20.827398    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 08:53:20.827840 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:20.827454    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 08:53:22.093146 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:22.093116    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:22.095098 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:22.095077    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:22.095810 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:22.095790    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 08:53:31.282052 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:53:31.276208    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:53:32.548909 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:32.548169    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.663946 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:32.661142    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.663946 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:32.661545    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 01 08:53:32.663946 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:32.663319    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.700258 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:32.700181    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:32.782003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:32.781953    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:33.554054 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:33.553126    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:33.570053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:33.554909    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:53:33.854633 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:33.854576    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:53:33.867056 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:33.859909    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:53:34.559191 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:34.559141    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:34.863143 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:34.861773    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:53:34.902224 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:34.902163    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:34.926162 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:34.926121    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:34.927764 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:34.927733    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 08:53:35.062169 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:35.062116    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:35.561051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:35.560754    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:35.742983 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:35.741278    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:53:35.750726 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:35.750691    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:53:52.575221 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:52.575120    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:52.669165 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:52.669117    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:52.671005 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:52.670414    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 08:53:52.789189 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:52.789142    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:52.899053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:52.876795    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:52.899053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:52.878751    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.187121 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.186930    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:53:53.187121 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.186997    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.610229 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.564377    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"]
Nov 01 08:53:53.653267 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.576311    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:53:53.710763 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.710732    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"]
Nov 01 08:53:53.793219 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.793190    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:53:53.793736 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.793716    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:53:53.802077 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.718512    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.898061 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.876939    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8971d8cb-bc36-424c-8523-3b31df38f235-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.900420 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.900388    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.900692 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.900672    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8971d8cb-bc36-424c-8523-3b31df38f235-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.900901 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.900872    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.901116 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.901099    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.901648 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.901628    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4n5n\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-kube-api-access-b4n5n\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.901957 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.901937    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.926053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.902218    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:53.949051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:53.932123    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.052840 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.052793    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.053151 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.053124    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.053359 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.053341    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8971d8cb-bc36-424c-8523-3b31df38f235-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.062060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.053512    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.062301 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.062275    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8971d8cb-bc36-424c-8523-3b31df38f235-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.062431 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.062412    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.062562 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.062546    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.075230 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.075183    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4n5n\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-kube-api-access-b4n5n\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.093213 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.093167    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.115095 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.091376    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.145477 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.144572    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.146669 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.091678    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.355080 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.326719    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.426128 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.417469    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.426128 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.418573    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8971d8cb-bc36-424c-8523-3b31df38f235-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.505051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.504491    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.610634 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.610587    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8971d8cb-bc36-424c-8523-3b31df38f235-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.668671 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.668461    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4n5n\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-kube-api-access-b4n5n\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.701639 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.701554    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:54.872332 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:54.872290    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:53:55.442380 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:55.442342    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"]
Nov 01 08:53:55.490839 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:55.490312    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:53:55.490981 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:55.490944    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:53:55.491079 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:55.491013    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:53:55.692277 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:55.692233    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" event={"ID":"8971d8cb-bc36-424c-8523-3b31df38f235","Type":"ContainerStarted","Data":"7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6"}
Nov 01 08:53:55.692791 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:55.692307    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" event={"ID":"8971d8cb-bc36-424c-8523-3b31df38f235","Type":"ContainerStarted","Data":"b5c5a018b5b319678dc64656a3f9b61332f1e07aae94f4bbe0e8fa0bbb4f01f7"}
Nov 01 08:53:55.700687 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:55.700661    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:53:55.810712 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:55.810664    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podStartSLOduration=2.810649917 podStartE2EDuration="2.810649917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:55.80835591 +0000 UTC m=+1305.923228877" watchObservedRunningTime="2026-11-01 08:53:55.810649917 +0000 UTC m=+1305.925522884"
Nov 01 08:53:56.701352 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:56.701317    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:53:56.701775 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:56.701393    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 08:53:57.700616 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:57.700580    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:53:57.700813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:57.700641    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 08:53:58.700450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:58.700415    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:53:58.701172 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:58.701134    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 08:53:59.700128 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:59.700087    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:53:59.700331 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:53:59.700161    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 08:54:00.706179 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:00.706144    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:54:00.706777 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:00.706748    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 08:54:01.023270 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:54:01.023178    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:54:01.701039 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:01.700983    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:54:01.701255 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:01.701082    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 08:54:02.700749 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:02.700712    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 08:54:02.701275 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:02.700786    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 08:54:03.706062 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:03.706015    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:54:03.706502 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:03.706328    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:54:03.707341 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:03.707314    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 08:54:09.667633 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:09.667592    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 08:54:09.669249 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:09.668196    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.784233 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:09.784174    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.877059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:09.876221    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:09.877059 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:09.876675    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 08:54:29.783199 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:29.783136    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:29.869173 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:29.869129    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:29.869557 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:29.869435    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:30.567230 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:30.567174    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:54:30.704191 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:30.682063    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 08:54:30.704191 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:30.687200    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 08:54:30.938060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:30.932491    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:30.938060 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:30.933399    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:54:31.506053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:31.489802    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.689057 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:31.688121    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:31.978057 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:31.975504    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.002100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:31.990766    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.400253 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:32.400180    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.473972 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:54:32.417820    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:54:32.632921 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:32.632183    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.671878 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:32.671136    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.671878 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:32.671537    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 08:54:32.696769 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:32.696730    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:32.697851 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:32.697828    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:54:32.797050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:32.793292    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.076117 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.045552    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.078400 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.048948    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 08:54:33.079872 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.078909    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:54:33.524577 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.501182    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:54:33.663084 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.663011    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.667303 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.667276    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:54:33.673637 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.673591    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.680164 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.680128    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 08:54:33.690203 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.681245    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:54:33.690203 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.681321    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:54:33.825345 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.794989    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:33.825345 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.795466    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:54:33.830504 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.830478    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:54:33.832362 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.832223    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:54:33.914050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.888498    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:54:33.949516 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.949489    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:54:33.952785 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.951300    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:54:33.957063 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:33.956489    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:54:41.317632 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.317591    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"]
Nov 01 08:54:41.321458 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.321435    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.374174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.374144    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-dshm\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.374362 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.374335    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkjd\" (UniqueName: \"kubernetes.io/projected/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-kube-api-access-vqkjd\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.374477 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.374423    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-model-cache\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.374649 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.374631    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tls-certs\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.374713 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.374685    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tmp-dir\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.374819 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.374801    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-home\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.420469 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.420441    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"distinct-2-w3-54e01b-kserve-self-signed-certs\""
Nov 01 08:54:41.423808 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.423785    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"]
Nov 01 08:54:41.477353 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.477316    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tmp-dir\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.477609 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.477584    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-home\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.477696 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.477628    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-dshm\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.477749 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.477720    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkjd\" (UniqueName: \"kubernetes.io/projected/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-kube-api-access-vqkjd\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.477821 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.477793    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-model-cache\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.477949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.477928    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tls-certs\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.478500 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.478476    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-home\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.487080 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.478744    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tmp-dir\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.487297 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.481881    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tls-certs\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.487297 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.482652    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-model-cache\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.487445 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.486554    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-dshm\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.542156 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.542116    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkjd\" (UniqueName: \"kubernetes.io/projected/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-kube-api-access-vqkjd\") pod \"distinct-2-w3-54e01b-kserve-858467db6b-9mzl8\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") " pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:41.634174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:41.634132    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:42.081543 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:54:42.081514    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5496b80d_d6ec_4409_8b8e_eea80d5b1d01.slice/crio-ea9b78b60274e733271291b13f2eaa2463ffcd922e18cfd585f953763b44dd96 WatchSource:0}: Error finding container ea9b78b60274e733271291b13f2eaa2463ffcd922e18cfd585f953763b44dd96: Status 404 returned error can't find the container with id ea9b78b60274e733271291b13f2eaa2463ffcd922e18cfd585f953763b44dd96
Nov 01 08:54:42.081669 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:42.081581    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"]
Nov 01 08:54:42.152177 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:42.152139    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" event={"ID":"5496b80d-d6ec-4409-8b8e-eea80d5b1d01","Type":"ContainerStarted","Data":"a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493"}
Nov 01 08:54:42.152359 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:42.152200    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" event={"ID":"5496b80d-d6ec-4409-8b8e-eea80d5b1d01","Type":"ContainerStarted","Data":"ea9b78b60274e733271291b13f2eaa2463ffcd922e18cfd585f953763b44dd96"}
Nov 01 08:54:42.266941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:42.266885    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podStartSLOduration=1.266868337 podStartE2EDuration="1.266868337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:42.263565066 +0000 UTC m=+1352.378438035" watchObservedRunningTime="2026-11-01 08:54:42.266868337 +0000 UTC m=+1352.381741302"
Nov 01 08:54:42.634646 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:42.634611    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:51.635221 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:51.635183    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:51.651830 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:51.651804    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:52.243333 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:52.243301    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:54.485071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.484114    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.647201 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.647160    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:54:54.647508 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.647476    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.834337 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.833112    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.834869 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.834581    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:54:54.908234 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.908178    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.972508 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.970116    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:54.972508 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.970505    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 01 08:54:54.997162 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:54.997116    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.239938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:55.239880    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.505106 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:55.502237    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.505106 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:55.503047    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:54:55.993112 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:55.837709    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:55.993112 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:55.838150    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:54:55.993112 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:55.839252    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:54:55.993112 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:55.867606    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:54:56.069001 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.047434    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:56.069001 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.058229    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:54:56.069794 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.069495    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:56.079446 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.072297    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.151185 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.075251    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.183157 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.183121    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:54:56.187049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.183496    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:54:56.278170 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.258847    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.348052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.341063    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:54:56.348052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.341187    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:56.348052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.341222    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 08:54:56.864634 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.863367    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:56.880062 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.871560    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:56.942352 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.942305    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 08:54:56.952047 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:56.947873    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.098226 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.098179    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.175104 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.154272    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.175104 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.154609    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Nov 01 08:54:57.198146 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.198095    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.253200 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.245201    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 08:54:57.502428 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.502320    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.557051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.526986    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.575051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.558050    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 08:54:57.581069 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.581041    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 08:54:57.607745 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.607701    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:57.963082 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.959107    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 08:54:57.988586 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.985613    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 08:54:58.026083 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:57.998529    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.179167 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:58.179118    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:54:58.550820 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:58.550791    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 08:54:58.551159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:54:58.551142    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 08:55:01.028590 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:55:01.028550    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:55:10.697753 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:10.697722    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"]
Nov 01 08:55:10.698260 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:10.697955    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy" containerID="cri-o://890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c" gracePeriod=30
Nov 01 08:55:15.975594 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:15.975538    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:55:16.152856 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.152823    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-credential-socket\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.153068 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.152914    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-envoy\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.153068 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.152943    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-token\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-token\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.153206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153067    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798f2b4f-841f-457f-b97d-75e41c4f2749-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/798f2b4f-841f-457f-b97d-75e41c4f2749-istiod-ca-cert\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.153206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153118    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-socket\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.153311 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153234    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-podinfo\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.153311 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153249    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-credential-socket" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:55:16.153311 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153295    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-certs\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.153461 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153374    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-kube-api-access-dn2wz\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-kube-api-access-dn2wz\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.153461 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153447    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-data\") pod \"798f2b4f-841f-457f-b97d-75e41c4f2749\" (UID: \"798f2b4f-841f-457f-b97d-75e41c4f2749\") "
Nov 01 08:55:16.154005 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153985    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.154191 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.153876    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798f2b4f-841f-457f-b97d-75e41c4f2749-istiod-ca-cert" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:55:16.154450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.154430    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-data" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:55:16.154593 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.154555    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-certs" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:55:16.154816 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.154793    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-socket" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:55:16.156183 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.156153    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-envoy" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 08:55:16.156474 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.156450    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-token" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:55:16.157489 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.157468    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-kube-api-access-dn2wz" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "kube-api-access-dn2wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:55:16.157489 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.157475    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-podinfo" pod "798f2b4f-841f-457f-b97d-75e41c4f2749" (UID: "798f2b4f-841f-457f-b97d-75e41c4f2749"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 08:55:16.255941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.255853    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.255941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.255884    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.255941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.255896    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn2wz\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-kube-api-access-dn2wz\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.255941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.255906    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.255941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.255914    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.255941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.255922    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/798f2b4f-841f-457f-b97d-75e41c4f2749-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.255941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.255930    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/798f2b4f-841f-457f-b97d-75e41c4f2749-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.255941 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.255939    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/798f2b4f-841f-457f-b97d-75e41c4f2749-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:55:16.694151 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.694110    4931 generic.go:356] "Generic (PLEG): container finished" podID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerID="890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c" exitCode=0
Nov 01 08:55:16.694416 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.694159    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"
Nov 01 08:55:16.694416 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.694194    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" event={"ID":"798f2b4f-841f-457f-b97d-75e41c4f2749","Type":"ContainerDied","Data":"890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c"}
Nov 01 08:55:16.694416 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.694251    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6" event={"ID":"798f2b4f-841f-457f-b97d-75e41c4f2749","Type":"ContainerDied","Data":"faf1e37090b2ca78b903869041549efb2f87910c907c2dc84faf06a1f41a8686"}
Nov 01 08:55:16.694416 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.694282    4931 scope.go:122] "RemoveContainer" containerID="890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c"
Nov 01 08:55:16.705103 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.705082    4931 scope.go:122] "RemoveContainer" containerID="890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c"
Nov 01 08:55:16.705399 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:55:16.705377    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c\": container with ID starting with 890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c not found: ID does not exist" containerID="890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c"
Nov 01 08:55:16.705445 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.705411    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c"} err="failed to get container status \"890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c\": rpc error: code = NotFound desc = could not find container \"890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c\": container with ID starting with 890de97e227ad1c0b53a8ecbe3bb946bc805c5530e4f34d0b38c3bd6afbdf34c not found: ID does not exist"
Nov 01 08:55:16.902591 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.902553    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"]
Nov 01 08:55:16.950460 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:16.950370    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-drdx6"]
Nov 01 08:55:18.055915 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:18.052165    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" path="/var/lib/kubelet/pods/798f2b4f-841f-457f-b97d-75e41c4f2749/volumes"
Nov 01 08:55:31.078112 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:55:31.075867    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:55:44.235070 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.235009    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"]
Nov 01 08:55:44.235884 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.235861    4931 cpu_manager.go:414] "containerMap: removing container" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy"
Nov 01 08:55:44.235884 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.235885    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy"
Nov 01 08:55:44.236088 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.236060    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="798f2b4f-841f-457f-b97d-75e41c4f2749" containerName="istio-proxy"
Nov 01 08:55:44.239671 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.239652    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.273964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.273927    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23d00cb-4858-40a3-baf4-6e1429ae747b-openshift-service-ca\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.273964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.273968    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p78c\" (UniqueName: \"kubernetes.io/projected/e23d00cb-4858-40a3-baf4-6e1429ae747b-kube-api-access-6p78c\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.274263 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.274142    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-metrics-certs\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.274333 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.274263    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.337270 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.337235    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-93665b-serving-cert\""
Nov 01 08:55:44.340833 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.340807    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"]
Nov 01 08:55:44.375932 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.375897    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-metrics-certs\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.376146 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.375993    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.376220 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.376195    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23d00cb-4858-40a3-baf4-6e1429ae747b-openshift-service-ca\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.376220 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:55:44.376206    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-serving-cert: secret "maas-api-e2e-api-a-93665b-serving-cert" not found
Nov 01 08:55:44.376330 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.376225    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p78c\" (UniqueName: \"kubernetes.io/projected/e23d00cb-4858-40a3-baf4-6e1429ae747b-kube-api-access-6p78c\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.376509 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:55:44.376491    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls podName:e23d00cb-4858-40a3-baf4-6e1429ae747b nodeName:}" failed. No retries permitted until 2026-11-01 08:55:44.876375734 +0000 UTC m=+1414.991248702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls") pod "maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc" (UID: "e23d00cb-4858-40a3-baf4-6e1429ae747b") : secret "maas-api-e2e-api-a-93665b-serving-cert" not found
Nov 01 08:55:44.377425 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.377394    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23d00cb-4858-40a3-baf4-6e1429ae747b-openshift-service-ca\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.379756 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.379731    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-metrics-certs\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.437006 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.436974    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p78c\" (UniqueName: \"kubernetes.io/projected/e23d00cb-4858-40a3-baf4-6e1429ae747b-kube-api-access-6p78c\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.884777 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.884743    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:44.887368 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:44.887342    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls\") pod \"maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:45.154701 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:45.154610    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:45.531312 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:45.531283    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"]
Nov 01 08:55:45.531682 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:55:45.531370    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23d00cb_4858_40a3_baf4_6e1429ae747b.slice/crio-a9b654dfbeef5e027d16d0dd28bde1f89ce658a7d13f4d673172bcd353e62a72 WatchSource:0}: Error finding container a9b654dfbeef5e027d16d0dd28bde1f89ce658a7d13f4d673172bcd353e62a72: Status 404 returned error can't find the container with id a9b654dfbeef5e027d16d0dd28bde1f89ce658a7d13f4d673172bcd353e62a72
Nov 01 08:55:45.532844 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:45.532823    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 08:55:45.946923 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:45.946890    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc" event={"ID":"e23d00cb-4858-40a3-baf4-6e1429ae747b","Type":"ContainerStarted","Data":"a9b654dfbeef5e027d16d0dd28bde1f89ce658a7d13f4d673172bcd353e62a72"}
Nov 01 08:55:46.965605 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:46.965565    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc" event={"ID":"e23d00cb-4858-40a3-baf4-6e1429ae747b","Type":"ContainerStarted","Data":"b554d40b1774e92d2848070402fea4dacbeb8996e26a60561a8831627658bcba"}
Nov 01 08:55:46.966107 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:46.965679    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:55:47.082791 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:47.082724    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc" podStartSLOduration=2.13840915 podStartE2EDuration="3.082709894s" totalImagesPullingTime="944.300744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:55:47.08042441 +0000 UTC m=+1417.195297404" watchObservedRunningTime="2026-11-01 08:55:47.082709894 +0000 UTC m=+1417.197582865"
Nov 01 08:55:52.978174 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:55:52.978146    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:56:01.028099 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:56:01.028063    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:56:01.302273 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.302182    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"]
Nov 01 08:56:01.306238 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.306216    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.404652 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.404615    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"distinct-w0-751a73-kserve-self-signed-certs\""
Nov 01 08:56:01.408264 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.408235    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"]
Nov 01 08:56:01.408548 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.408529    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tls-certs\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.408615 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.408600    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94sx7\" (UniqueName: \"kubernetes.io/projected/26fa7b17-f3ec-4165-b22a-09c4d95837d6-kube-api-access-94sx7\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.408671 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.408662    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-model-cache\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.408732 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.408684    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tmp-dir\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.408786 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.408732    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-home\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.408840 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.408812    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-dshm\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.509964 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.509927    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94sx7\" (UniqueName: \"kubernetes.io/projected/26fa7b17-f3ec-4165-b22a-09c4d95837d6-kube-api-access-94sx7\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.510274 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.510253    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-model-cache\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.510462 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.510444    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tmp-dir\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.511804 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.511648    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-model-cache\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.511938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.511656    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tmp-dir\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.512088 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.512066    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-home\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.512227 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.512204    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-dshm\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.512488 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.512463    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tls-certs\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.513038 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.513002    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-home\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.520328 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.516482    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tls-certs\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.524451 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.524426    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-dshm\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.572051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.570337    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94sx7\" (UniqueName: \"kubernetes.io/projected/26fa7b17-f3ec-4165-b22a-09c4d95837d6-kube-api-access-94sx7\") pod \"distinct-w0-751a73-kserve-79f86d5649-2gdqk\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:01.618586 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:01.618489    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:02.156409 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:02.156375    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"]
Nov 01 08:56:02.169051 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:56:02.168994    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26fa7b17_f3ec_4165_b22a_09c4d95837d6.slice/crio-1c79b21e30ea115646947d45a0d2eac36ab20b441101d0651a11535ffee40a37 WatchSource:0}: Error finding container 1c79b21e30ea115646947d45a0d2eac36ab20b441101d0651a11535ffee40a37: Status 404 returned error can't find the container with id 1c79b21e30ea115646947d45a0d2eac36ab20b441101d0651a11535ffee40a37
Nov 01 08:56:03.103136 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:03.103093    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" event={"ID":"26fa7b17-f3ec-4165-b22a-09c4d95837d6","Type":"ContainerStarted","Data":"23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef"}
Nov 01 08:56:03.103309 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:03.103150    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" event={"ID":"26fa7b17-f3ec-4165-b22a-09c4d95837d6","Type":"ContainerStarted","Data":"1c79b21e30ea115646947d45a0d2eac36ab20b441101d0651a11535ffee40a37"}
Nov 01 08:56:03.218802 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:03.218754    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podStartSLOduration=2.218739418 podStartE2EDuration="2.218739418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:03.21815577 +0000 UTC m=+1433.333028738" watchObservedRunningTime="2026-11-01 08:56:03.218739418 +0000 UTC m=+1433.333612386"
Nov 01 08:56:03.618812 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:03.618780    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:11.618987 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:11.618949    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:11.631893 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:11.631868    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:12.181082 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:12.181053    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 08:56:29.196110 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.196074    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"]
Nov 01 08:56:29.200459 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.200436    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.221648 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.221613    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-dshm\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.221826 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.221655    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwdfm\" (UniqueName: \"kubernetes.io/projected/5405ff20-f30b-4c5c-818f-b5ab5a06a940-kube-api-access-dwdfm\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.221926 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.221833    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tls-certs\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.222018 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.221996    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-home\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.222206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.222189    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-model-cache\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.222298 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.222220    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tmp-dir\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.259424 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.259388    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"]
Nov 01 08:56:29.300165 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.300130    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d31acd45\"/\"kube-root-ca.crt\""
Nov 01 08:56:29.300377 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.300264    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d31acd45\"/\"openshift-service-ca.crt\""
Nov 01 08:56:29.300377 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.300270    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d31acd45\"/\"default-dockercfg-kr2x9\""
Nov 01 08:56:29.300377 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.300329    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d31acd45\"/\"e2e-lifecycle-model-d31acd45-kserve-self-signed-certs\""
Nov 01 08:56:29.323775 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.323740    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tls-certs\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.323949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.323845    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-home\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.323949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.323910    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-model-cache\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.323949 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.323932    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tmp-dir\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.324133 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.324061    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-dshm\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.324133 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.324086    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwdfm\" (UniqueName: \"kubernetes.io/projected/5405ff20-f30b-4c5c-818f-b5ab5a06a940-kube-api-access-dwdfm\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.324650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.324611    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tmp-dir\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.324650 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.324639    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-model-cache\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.324868 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.324847    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-home\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.326561 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.326539    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tls-certs\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.326671 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.326603    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-dshm\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.389942 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.389908    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwdfm\" (UniqueName: \"kubernetes.io/projected/5405ff20-f30b-4c5c-818f-b5ab5a06a940-kube-api-access-dwdfm\") pod \"e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") " pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.512772 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.512677    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:29.894995 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:56:29.894959    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5405ff20_f30b_4c5c_818f_b5ab5a06a940.slice/crio-fb65c96488d01c027eaebbd92ef45be3e1bd3b96b127964ebdec48735239f9f2 WatchSource:0}: Error finding container fb65c96488d01c027eaebbd92ef45be3e1bd3b96b127964ebdec48735239f9f2: Status 404 returned error can't find the container with id fb65c96488d01c027eaebbd92ef45be3e1bd3b96b127964ebdec48735239f9f2
Nov 01 08:56:29.896367 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:29.896338    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"]
Nov 01 08:56:30.337189 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:30.337087    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6" event={"ID":"5405ff20-f30b-4c5c-818f-b5ab5a06a940","Type":"ContainerStarted","Data":"0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a"}
Nov 01 08:56:30.337189 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:30.337153    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6" event={"ID":"5405ff20-f30b-4c5c-818f-b5ab5a06a940","Type":"ContainerStarted","Data":"fb65c96488d01c027eaebbd92ef45be3e1bd3b96b127964ebdec48735239f9f2"}
Nov 01 08:56:30.452042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:30.451974    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6" podStartSLOduration=1.451960343 podStartE2EDuration="1.451960343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:30.448972616 +0000 UTC m=+1460.563845585" watchObservedRunningTime="2026-11-01 08:56:30.451960343 +0000 UTC m=+1460.566833359"
Nov 01 08:56:30.513413 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:30.513381    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:31.023886 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:56:31.023849    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:56:39.381048 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.377941    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"]
Nov 01 08:56:39.385586 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.385565    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.432762 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.432727    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"]
Nov 01 08:56:39.478931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.468916    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.478931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.469124    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22f2095-450d-4762-b582-5c12007fa190-openshift-service-ca\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.478931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.469158    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-metrics-certs\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.478931 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.469250    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj965\" (UniqueName: \"kubernetes.io/projected/a22f2095-450d-4762-b582-5c12007fa190-kube-api-access-gj965\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.486095 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.483288    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5fad29-serving-cert\""
Nov 01 08:56:39.513320 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.513285    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:39.526457 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.526427    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:39.570654 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.570623    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.570850 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:56:39.570777    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-serving-cert: secret "maas-api-e2e-api-a-5fad29-serving-cert" not found
Nov 01 08:56:39.570850 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.570818    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22f2095-450d-4762-b582-5c12007fa190-openshift-service-ca\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.570850 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:56:39.570846    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls podName:a22f2095-450d-4762-b582-5c12007fa190 nodeName:}" failed. No retries permitted until 2026-11-01 08:56:40.07082337 +0000 UTC m=+1470.185696339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls") pod "maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk" (UID: "a22f2095-450d-4762-b582-5c12007fa190") : secret "maas-api-e2e-api-a-5fad29-serving-cert" not found
Nov 01 08:56:39.571051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.570881    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-metrics-certs\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.571051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.571012    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj965\" (UniqueName: \"kubernetes.io/projected/a22f2095-450d-4762-b582-5c12007fa190-kube-api-access-gj965\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.571606 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.571588    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22f2095-450d-4762-b582-5c12007fa190-openshift-service-ca\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.573646 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.573621    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-metrics-certs\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:39.631300 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:39.631230    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj965\" (UniqueName: \"kubernetes.io/projected/a22f2095-450d-4762-b582-5c12007fa190-kube-api-access-gj965\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:40.084127 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:40.079011    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:40.084127 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:40.082589    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls\") pod \"maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:40.308056 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:40.306927    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:40.477431 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:40.476636    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 08:56:40.742214 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:40.742184    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"]
Nov 01 08:56:41.404051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.403287    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"]
Nov 01 08:56:41.415590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.415558    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440119    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440156    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440189    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440287    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440452    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440490    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edf08ed-02ef-4a05-9abf-dbe60053f296-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440574    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440607    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.441042 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.440632    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdr75\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-kube-api-access-sdr75\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.443072 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.441924    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk" event={"ID":"a22f2095-450d-4762-b582-5c12007fa190","Type":"ContainerStarted","Data":"1d0442b867b7b11a2e8dc492468d36239cb12c8444c1d0a4ff8512a823edcfc2"}
Nov 01 08:56:41.474051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.468206    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-751a73-openshift-default-dockercfg-qq2cf\""
Nov 01 08:56:41.534618 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.534576    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"]
Nov 01 08:56:41.542537 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542506    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.542717 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542550    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.542717 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542614    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.542717 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542635    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edf08ed-02ef-4a05-9abf-dbe60053f296-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.542717 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542676    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.542717 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542697    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.542717 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542715    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdr75\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-kube-api-access-sdr75\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.543073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542817    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.543073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542838    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.543073 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.542854    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.543331 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.543308    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.543996 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.543921    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.544170 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.543982    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.544853 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.544837    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edf08ed-02ef-4a05-9abf-dbe60053f296-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.545994 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.545966    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.546592 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.546567    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.647194 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.646665    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.653817 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.653743    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdr75\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-kube-api-access-sdr75\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:41.758207 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:41.749941    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:42.640363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:42.620200    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk" event={"ID":"a22f2095-450d-4762-b582-5c12007fa190","Type":"ContainerStarted","Data":"fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314"}
Nov 01 08:56:42.640363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:42.621284    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:42.854063 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:42.822928    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk" podStartSLOduration=2.6393095989999997 podStartE2EDuration="3.822908379s" totalImagesPullingTime="1.18359878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:42.809649213 +0000 UTC m=+1472.924522182" watchObservedRunningTime="2026-11-01 08:56:42.822908379 +0000 UTC m=+1472.937781347"
Nov 01 08:56:43.020077 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:42.998316    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"]
Nov 01 08:56:43.040051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:43.026246    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:56:43.040051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:43.026345    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:56:43.040051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:43.026388    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:56:43.044053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:43.043143    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:56:43.648051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:43.638791    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" event={"ID":"0edf08ed-02ef-4a05-9abf-dbe60053f296","Type":"ContainerStarted","Data":"490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78"}
Nov 01 08:56:43.648051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:43.638848    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" event={"ID":"0edf08ed-02ef-4a05-9abf-dbe60053f296","Type":"ContainerStarted","Data":"553484d89e06a0ddd36dc67f795566c73fd5e5cc17525ff32d16bece76487d3d"}
Nov 01 08:56:43.750786 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:43.750757    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:43.807053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:43.790769    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podStartSLOduration=2.790744294 podStartE2EDuration="2.790744294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:43.76934595 +0000 UTC m=+1473.884218919" watchObservedRunningTime="2026-11-01 08:56:43.790744294 +0000 UTC m=+1473.905617266"
Nov 01 08:56:44.752954 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:44.752916    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:56:44.753618 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:44.752975    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 01 08:56:45.750771 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:45.750735    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:56:45.750984 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:45.750787    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 01 08:56:46.750912 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:46.750877    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:56:46.751364 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:46.750939    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 01 08:56:47.751001 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:47.750969    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:56:47.751437 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:47.751059    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 01 08:56:48.751590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:48.751553    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:56:48.752006 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:48.751612    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 01 08:56:49.690286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:49.690198    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 08:56:49.750937 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:49.750905    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:56:49.751141 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:49.750956    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 01 08:56:50.751016 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:50.750969    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 01 08:56:50.751462 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:50.751083    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 01 08:56:51.756014 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:51.755980    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:51.756452 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:51.756320    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:51.757086 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:51.757066    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 08:56:54.723072 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.723039    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"]
Nov 01 08:56:54.727915 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.727899    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.783624 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.783588    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"]
Nov 01 08:56:54.794897 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.794861    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tmp-dir\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.795096 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.794921    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-model-cache\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.795096 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.794969    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tls-certs\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.795240 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.795217    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptj9n\" (UniqueName: \"kubernetes.io/projected/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-kube-api-access-ptj9n\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.795434 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.795403    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-dshm\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.795500 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.795478    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-home\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.825591 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.825554    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-751a73\"/\"distinct-2-w0-751a73-kserve-self-signed-certs\""
Nov 01 08:56:54.896666 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.896624    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-home\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.896871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.896744    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tmp-dir\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.896871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.896786    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-model-cache\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.896871 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.896833    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tls-certs\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.897100 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.896958    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptj9n\" (UniqueName: \"kubernetes.io/projected/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-kube-api-access-ptj9n\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.897507 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.897120    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-dshm\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.897507 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.897123    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-home\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.897507 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.897166    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tmp-dir\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.897507 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.897261    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-model-cache\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.899847 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.899824    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-dshm\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.901630 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.901608    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tls-certs\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:54.957685 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:54.957642    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptj9n\" (UniqueName: \"kubernetes.io/projected/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-kube-api-access-ptj9n\") pod \"distinct-2-w0-751a73-kserve-8677d98654-kcf8k\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") " pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:55.041499 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:55.041415    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:56:55.489315 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:55.489285    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"]
Nov 01 08:56:55.489476 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:56:55.489458    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd1ce16d_b00e_4a68_8b2b_7e3ec9a6fa49.slice/crio-b73943f138dbd259ce8fd08019bc376e03837d81561993113ad66456e4754b1a WatchSource:0}: Error finding container b73943f138dbd259ce8fd08019bc376e03837d81561993113ad66456e4754b1a: Status 404 returned error can't find the container with id b73943f138dbd259ce8fd08019bc376e03837d81561993113ad66456e4754b1a
Nov 01 08:56:55.765575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:55.765475    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k" event={"ID":"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49","Type":"ContainerStarted","Data":"ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab"}
Nov 01 08:56:55.765575 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:55.765520    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k" event={"ID":"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49","Type":"ContainerStarted","Data":"b73943f138dbd259ce8fd08019bc376e03837d81561993113ad66456e4754b1a"}
Nov 01 08:56:55.882127 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:55.882074    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k" podStartSLOduration=1.882059039 podStartE2EDuration="1.882059039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:55.880309208 +0000 UTC m=+1485.995182175" watchObservedRunningTime="2026-11-01 08:56:55.882059039 +0000 UTC m=+1485.996932006"
Nov 01 08:56:56.049859 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:56:56.049773    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:57:01.028678 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:57:01.028634    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:57:05.042069 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:05.042009    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:57:05.054877 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:05.054857    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:57:05.847196 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:05.847166    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 08:57:10.223829 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:10.223800    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:57:10.227298 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:10.227270    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 08:57:10.227927 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:10.227910    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:57:10.231458 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:10.231439    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 08:57:23.922049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:23.921990    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"]
Nov 01 08:57:23.927691 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:23.927674    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:23.981637 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:23.981607    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"]
Nov 01 08:57:24.013300 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.013257    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glftc\" (UniqueName: \"kubernetes.io/projected/8a68e294-5862-46bd-a1f9-c41ae87e3c94-kube-api-access-glftc\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.013300 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.013309    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.013569 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.013493    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68e294-5862-46bd-a1f9-c41ae87e3c94-openshift-service-ca\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.013569 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.013543    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-metrics-certs\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.024827 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.024782    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8701a0-serving-cert\""
Nov 01 08:57:24.115056 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.114995    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-metrics-certs\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.115234 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.115184    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glftc\" (UniqueName: \"kubernetes.io/projected/8a68e294-5862-46bd-a1f9-c41ae87e3c94-kube-api-access-glftc\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.115234 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.115217    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.115312 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.115285    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68e294-5862-46bd-a1f9-c41ae87e3c94-openshift-service-ca\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.115431 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:57:24.115413    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-serving-cert: secret "maas-api-e2e-api-b-8701a0-serving-cert" not found
Nov 01 08:57:24.115514 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:57:24.115501    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls podName:8a68e294-5862-46bd-a1f9-c41ae87e3c94 nodeName:}" failed. No retries permitted until 2026-11-01 08:57:24.615479291 +0000 UTC m=+1514.730352254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls") pod "maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7" (UID: "8a68e294-5862-46bd-a1f9-c41ae87e3c94") : secret "maas-api-e2e-api-b-8701a0-serving-cert" not found
Nov 01 08:57:24.115979 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.115962    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68e294-5862-46bd-a1f9-c41ae87e3c94-openshift-service-ca\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.117676 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.117658    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-metrics-certs\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.180426 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.180292    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glftc\" (UniqueName: \"kubernetes.io/projected/8a68e294-5862-46bd-a1f9-c41ae87e3c94-kube-api-access-glftc\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.622838 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.622804    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.625570 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.625546    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls\") pod \"maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:24.840950 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:24.840918    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:25.216914 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:25.216882    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"]
Nov 01 08:57:25.217436 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:57:25.217008    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a68e294_5862_46bd_a1f9_c41ae87e3c94.slice/crio-8409b6a9415a6930a375d7b43446b285a22c76f853029a7fd26d043f72bd1dda WatchSource:0}: Error finding container 8409b6a9415a6930a375d7b43446b285a22c76f853029a7fd26d043f72bd1dda: Status 404 returned error can't find the container with id 8409b6a9415a6930a375d7b43446b285a22c76f853029a7fd26d043f72bd1dda
Nov 01 08:57:25.998527 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:25.998486    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7" event={"ID":"8a68e294-5862-46bd-a1f9-c41ae87e3c94","Type":"ContainerStarted","Data":"8409b6a9415a6930a375d7b43446b285a22c76f853029a7fd26d043f72bd1dda"}
Nov 01 08:57:26.754913 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.754877    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"]
Nov 01 08:57:26.759607 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.759587    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.858998 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.858961    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"]
Nov 01 08:57:26.861677 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.860055    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwv7d\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-kube-api-access-jwv7d\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.861932 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.861914    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.862143 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.862122    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b859b481-69c9-445b-b38f-aefd47600932-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.862356 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.862338    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b859b481-69c9-445b-b38f-aefd47600932-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.862571 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.862548    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.862689 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.862625    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.862760 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.862694    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.862813 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.862796    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.862874 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.862855    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.977352 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.969985    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.980166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.980127    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.980450 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.980433    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwv7d\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-kube-api-access-jwv7d\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.980590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.980576    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.981677 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.981655    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b859b481-69c9-445b-b38f-aefd47600932-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.984286 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.984261    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b859b481-69c9-445b-b38f-aefd47600932-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.984572 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.978755    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.984572 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.981573    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.986159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.984886    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.986159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.984981    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b859b481-69c9-445b-b38f-aefd47600932-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.986159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.984988    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.986159 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.985129    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.987209 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.986857    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.987209 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.987130    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.988081 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.988005    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b859b481-69c9-445b-b38f-aefd47600932-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:26.988549 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:26.988528    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:27.010624 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.010551    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7" event={"ID":"8a68e294-5862-46bd-a1f9-c41ae87e3c94","Type":"ContainerStarted","Data":"6fbcbb7bfa7cdb443474416df7811a51ac8f5997a66e7829038a6dbcc5433ab6"}
Nov 01 08:57:27.010753 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.010673    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:27.040363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.040047    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:27.040363 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.040243    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwv7d\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-kube-api-access-jwv7d\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:27.073654 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.073618    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:27.125013 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.124953    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7" podStartSLOduration=3.254215843 podStartE2EDuration="4.124937015s" totalImagesPullingTime="870.721172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:27.122910328 +0000 UTC m=+1517.237783286" watchObservedRunningTime="2026-11-01 08:57:27.124937015 +0000 UTC m=+1517.239809999"
Nov 01 08:57:27.536920 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:57:27.536882    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb859b481_69c9_445b_b38f_aefd47600932.slice/crio-6b64454384fdcd8904534b60f9954b620985f6f7382e979c2e997ff32866612c WatchSource:0}: Error finding container 6b64454384fdcd8904534b60f9954b620985f6f7382e979c2e997ff32866612c: Status 404 returned error can't find the container with id 6b64454384fdcd8904534b60f9954b620985f6f7382e979c2e997ff32866612c
Nov 01 08:57:27.537888 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.537866    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"]
Nov 01 08:57:27.539538 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.539507    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:57:27.539611 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.539576    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:57:27.539611 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:27.539603    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:57:28.022461 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:28.022422    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" event={"ID":"b859b481-69c9-445b-b38f-aefd47600932","Type":"ContainerStarted","Data":"21ac25c6995972713c9730180a61d605c696b675e2798327f4464743edef9a69"}
Nov 01 08:57:28.022977 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:28.022489    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" event={"ID":"b859b481-69c9-445b-b38f-aefd47600932","Type":"ContainerStarted","Data":"6b64454384fdcd8904534b60f9954b620985f6f7382e979c2e997ff32866612c"}
Nov 01 08:57:28.083050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:28.079341    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:28.083050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:28.079545    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 08:57:28.083050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:28.079600    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 08:57:28.146644 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:28.146580    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podStartSLOduration=2.146558807 podStartE2EDuration="2.146558807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:28.142958202 +0000 UTC m=+1518.257831172" watchObservedRunningTime="2026-11-01 08:57:28.146558807 +0000 UTC m=+1518.261431776"
Nov 01 08:57:29.078050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:29.077461    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 08:57:29.078050 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:29.077546    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 08:57:30.084237 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:30.084197    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 08:57:30.084667 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:30.084309    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 08:57:31.024804 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:57:31.024769    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:57:31.080052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:31.074431    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 08:57:31.080052 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:31.074507    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 08:57:32.082501 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:32.082459    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 08:57:32.083104 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:32.083068    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 08:57:33.028826 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:33.028797    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 08:57:33.074554 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:33.074512    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 08:57:33.074862 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:33.074830    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 08:57:34.074378 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:34.074339    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 08:57:34.074809 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:34.074407    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 08:57:35.500927 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:35.500892    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:35.502304 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:35.502285    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:35.506483 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:35.506448    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 08:57:41.871521 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:41.871483    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"]
Nov 01 08:57:41.878083 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:41.878056    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:41.982694 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:41.982656    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"]
Nov 01 08:57:42.043150 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043108    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.043349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043225    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.043349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043267    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.043349 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043316    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgb7g\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-kube-api-access-lgb7g\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.043590 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043570    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.043627 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043611    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.043690 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043674    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a925225-2eca-43a5-b4da-67857dcc22a3-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.043736 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043722    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.043823 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.043808    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145481 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145397    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgb7g\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-kube-api-access-lgb7g\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145644 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145544    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145644 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145565    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145644 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145585    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a925225-2eca-43a5-b4da-67857dcc22a3-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145644 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145621    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145847 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145673    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145847 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145695    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145847 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145741    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.145847 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.145764    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.146107 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.146085    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.146175 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.146120    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.146232 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.146212    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.146269 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.146233    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.146458 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.146440    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a925225-2eca-43a5-b4da-67857dcc22a3-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.148774 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.148753    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.148974 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.148955    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.217251 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.217204    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.217432 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.217315    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgb7g\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-kube-api-access-lgb7g\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.498637 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.498548    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:42.952443 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:57:42.952407    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a925225_2eca_43a5_b4da_67857dcc22a3.slice/crio-306482324e663a55c31ae22e4fd6e2ee18f4b834e905d42cc6c2e34e55422379 WatchSource:0}: Error finding container 306482324e663a55c31ae22e4fd6e2ee18f4b834e905d42cc6c2e34e55422379: Status 404 returned error can't find the container with id 306482324e663a55c31ae22e4fd6e2ee18f4b834e905d42cc6c2e34e55422379
Nov 01 08:57:42.952934 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.952852    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"]
Nov 01 08:57:42.954998 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.954966    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:57:42.955081 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.955063    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:57:42.955117 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:42.955095    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 08:57:43.179821 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:43.179789    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" event={"ID":"4a925225-2eca-43a5-b4da-67857dcc22a3","Type":"ContainerStarted","Data":"c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e"}
Nov 01 08:57:43.180003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:43.179846    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" event={"ID":"4a925225-2eca-43a5-b4da-67857dcc22a3","Type":"ContainerStarted","Data":"306482324e663a55c31ae22e4fd6e2ee18f4b834e905d42cc6c2e34e55422379"}
Nov 01 08:57:43.297987 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:43.297871    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" podStartSLOduration=2.297857586 podStartE2EDuration="2.297857586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:43.295673867 +0000 UTC m=+1533.410546837" watchObservedRunningTime="2026-11-01 08:57:43.297857586 +0000 UTC m=+1533.412730554"
Nov 01 08:57:43.499206 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:43.499174    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:44.501691 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:44.501650    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:57:44.502120 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:44.501727    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 01 08:57:45.501066 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:45.499639    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:57:45.501066 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:45.499702    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 01 08:57:46.499288 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:46.499251    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:57:46.499721 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:46.499320    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 01 08:57:47.499638 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:47.499604    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:57:47.500066 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:47.499664    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 01 08:57:48.499350 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:48.499313    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:57:48.499540 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:48.499380    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 01 08:57:49.499543 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:49.499501    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 01 08:57:49.500049 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:49.499566    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 01 08:57:51.050383 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:51.050351    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:51.050851 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:51.050824    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:57:51.051648 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:57:51.051631    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 08:58:01.023634 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:58:01.023596    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:58:31.024630 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:58:31.024552    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:58:36.570051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:36.569214    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 08:58:36.570051 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:36.569283    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 08:58:51.027726 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.027692    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"]
Nov 01 08:58:51.032224 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.032203    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.082253 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.082224    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-03832c-serving-cert\""
Nov 01 08:58:51.122071 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.122001    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27669d73-86a4-4f64-8582-23ac0072cd81-openshift-service-ca\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.122248 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.122229    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.122409 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.122392    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-metrics-certs\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.122552 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.122534    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwrh\" (UniqueName: \"kubernetes.io/projected/27669d73-86a4-4f64-8582-23ac0072cd81-kube-api-access-ndwrh\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.133882 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.133851    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"]
Nov 01 08:58:51.224003 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.223972    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.224189 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.224110    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-metrics-certs\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.224189 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:58:51.224126    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-serving-cert: secret "maas-api-e2e-api-a-03832c-serving-cert" not found
Nov 01 08:58:51.224289 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:58:51.224192    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls podName:27669d73-86a4-4f64-8582-23ac0072cd81 nodeName:}" failed. No retries permitted until 2026-11-01 08:58:51.724171676 +0000 UTC m=+1601.839044623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls") pod "maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b" (UID: "27669d73-86a4-4f64-8582-23ac0072cd81") : secret "maas-api-e2e-api-a-03832c-serving-cert" not found
Nov 01 08:58:51.224289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.224237    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwrh\" (UniqueName: \"kubernetes.io/projected/27669d73-86a4-4f64-8582-23ac0072cd81-kube-api-access-ndwrh\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.224289 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.224279    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27669d73-86a4-4f64-8582-23ac0072cd81-openshift-service-ca\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.224908 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.224862    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27669d73-86a4-4f64-8582-23ac0072cd81-openshift-service-ca\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.226938 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.226915    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-metrics-certs\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.291177 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.291058    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwrh\" (UniqueName: \"kubernetes.io/projected/27669d73-86a4-4f64-8582-23ac0072cd81-kube-api-access-ndwrh\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.731479 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.731433    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.734400 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.734364    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls\") pod \"maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:51.946705 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:51.946675    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:52.326713 ip-10-0-143-244 kubenswrapper[4931]: W1101 08:58:52.326675    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27669d73_86a4_4f64_8582_23ac0072cd81.slice/crio-717eed4967c6d81129f5b3ed4b72a5d55c090267aeddd5900ff54af400f058f2 WatchSource:0}: Error finding container 717eed4967c6d81129f5b3ed4b72a5d55c090267aeddd5900ff54af400f058f2: Status 404 returned error can't find the container with id 717eed4967c6d81129f5b3ed4b72a5d55c090267aeddd5900ff54af400f058f2
Nov 01 08:58:52.327166 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:52.326938    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"]
Nov 01 08:58:52.856576 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:52.856541    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b" event={"ID":"27669d73-86a4-4f64-8582-23ac0072cd81","Type":"ContainerStarted","Data":"717eed4967c6d81129f5b3ed4b72a5d55c090267aeddd5900ff54af400f058f2"}
Nov 01 08:58:53.868709 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:53.868670    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b" event={"ID":"27669d73-86a4-4f64-8582-23ac0072cd81","Type":"ContainerStarted","Data":"4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe"}
Nov 01 08:58:53.869247 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:53.868747    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:58:53.984933 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:53.984842    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b" podStartSLOduration=3.064893101 podStartE2EDuration="3.984804292s" totalImagesPullingTime="919.911191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:58:53.981557817 +0000 UTC m=+1604.096430785" watchObservedRunningTime="2026-11-01 08:58:53.984804292 +0000 UTC m=+1604.099677259"
Nov 01 08:58:55.469717 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:55.469674    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"]
Nov 01 08:58:55.470235 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:55.469961    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc" podUID="e23d00cb-4858-40a3-baf4-6e1429ae747b" containerName="maas-api" containerID="cri-o://b554d40b1774e92d2848070402fea4dacbeb8996e26a60561a8831627658bcba" gracePeriod=30
Nov 01 08:58:55.911693 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:55.909683    4931 generic.go:356] "Generic (PLEG): container finished" podID="e23d00cb-4858-40a3-baf4-6e1429ae747b" containerID="b554d40b1774e92d2848070402fea4dacbeb8996e26a60561a8831627658bcba" exitCode=0
Nov 01 08:58:55.911693 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:55.909863    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc" event={"ID":"e23d00cb-4858-40a3-baf4-6e1429ae747b","Type":"ContainerDied","Data":"b554d40b1774e92d2848070402fea4dacbeb8996e26a60561a8831627658bcba"}
Nov 01 08:58:55.911693 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:55.909920    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc" event={"ID":"e23d00cb-4858-40a3-baf4-6e1429ae747b","Type":"ContainerDied","Data":"a9b654dfbeef5e027d16d0dd28bde1f89ce658a7d13f4d673172bcd353e62a72"}
Nov 01 08:58:55.911693 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:55.909954    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b654dfbeef5e027d16d0dd28bde1f89ce658a7d13f4d673172bcd353e62a72"
Nov 01 08:58:55.934053 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:55.925222    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:58:56.018232 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.014742    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls\") pod \"e23d00cb-4858-40a3-baf4-6e1429ae747b\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") "
Nov 01 08:58:56.018232 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.014832    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23d00cb-4858-40a3-baf4-6e1429ae747b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23d00cb-4858-40a3-baf4-6e1429ae747b-openshift-service-ca\") pod \"e23d00cb-4858-40a3-baf4-6e1429ae747b\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") "
Nov 01 08:58:56.018232 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.014859    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23d00cb-4858-40a3-baf4-6e1429ae747b-kube-api-access-6p78c\" (UniqueName: \"kubernetes.io/projected/e23d00cb-4858-40a3-baf4-6e1429ae747b-kube-api-access-6p78c\") pod \"e23d00cb-4858-40a3-baf4-6e1429ae747b\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") "
Nov 01 08:58:56.018232 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.015882    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-metrics-certs\") pod \"e23d00cb-4858-40a3-baf4-6e1429ae747b\" (UID: \"e23d00cb-4858-40a3-baf4-6e1429ae747b\") "
Nov 01 08:58:56.018232 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.015966    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23d00cb-4858-40a3-baf4-6e1429ae747b-openshift-service-ca" pod "e23d00cb-4858-40a3-baf4-6e1429ae747b" (UID: "e23d00cb-4858-40a3-baf4-6e1429ae747b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 08:58:56.018232 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.017655    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23d00cb-4858-40a3-baf4-6e1429ae747b-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:56.019089 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.018803    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-metrics-certs" pod "e23d00cb-4858-40a3-baf4-6e1429ae747b" (UID: "e23d00cb-4858-40a3-baf4-6e1429ae747b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:58:56.021129 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.021099    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls" pod "e23d00cb-4858-40a3-baf4-6e1429ae747b" (UID: "e23d00cb-4858-40a3-baf4-6e1429ae747b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 08:58:56.021513 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.021492    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23d00cb-4858-40a3-baf4-6e1429ae747b-kube-api-access-6p78c" pod "e23d00cb-4858-40a3-baf4-6e1429ae747b" (UID: "e23d00cb-4858-40a3-baf4-6e1429ae747b"). InnerVolumeSpecName "kube-api-access-6p78c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 08:58:56.119369 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.119335    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:56.119369 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.119371    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p78c\" (UniqueName: \"kubernetes.io/projected/e23d00cb-4858-40a3-baf4-6e1429ae747b-kube-api-access-6p78c\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:56.119657 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.119390    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e23d00cb-4858-40a3-baf4-6e1429ae747b-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 08:58:56.916316 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:56.916283    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"
Nov 01 08:58:57.077525 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:57.077488    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"]
Nov 01 08:58:57.123058 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:57.123011    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93665b-65d8f869fd-b7jdc"]
Nov 01 08:58:58.049005 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:58.048971    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23d00cb-4858-40a3-baf4-6e1429ae747b" path="/var/lib/kubelet/pods/e23d00cb-4858-40a3-baf4-6e1429ae747b/volumes"
Nov 01 08:58:59.883405 ip-10-0-143-244 kubenswrapper[4931]: I1101 08:58:59.883379    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 08:59:01.025983 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:59:01.025943    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 08:59:31.024608 ip-10-0-143-244 kubenswrapper[4931]: E1101 08:59:31.024567    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:00:01.029586 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:00:01.029494    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:00:04.190540 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.190505    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"]
Nov 01 09:00:04.190954 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.190727    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7" podUID="8a68e294-5862-46bd-a1f9-c41ae87e3c94" containerName="maas-api" containerID="cri-o://6fbcbb7bfa7cdb443474416df7811a51ac8f5997a66e7829038a6dbcc5433ab6" gracePeriod=30
Nov 01 09:00:04.527441 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.527340    4931 generic.go:356] "Generic (PLEG): container finished" podID="8a68e294-5862-46bd-a1f9-c41ae87e3c94" containerID="6fbcbb7bfa7cdb443474416df7811a51ac8f5997a66e7829038a6dbcc5433ab6" exitCode=0
Nov 01 09:00:04.527612 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.527545    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7" event={"ID":"8a68e294-5862-46bd-a1f9-c41ae87e3c94","Type":"ContainerDied","Data":"6fbcbb7bfa7cdb443474416df7811a51ac8f5997a66e7829038a6dbcc5433ab6"}
Nov 01 09:00:04.616354 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.616322    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 09:00:04.715342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.715304    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a68e294-5862-46bd-a1f9-c41ae87e3c94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68e294-5862-46bd-a1f9-c41ae87e3c94-openshift-service-ca\") pod \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") "
Nov 01 09:00:04.716134 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.716111    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a68e294-5862-46bd-a1f9-c41ae87e3c94-kube-api-access-glftc\" (UniqueName: \"kubernetes.io/projected/8a68e294-5862-46bd-a1f9-c41ae87e3c94-kube-api-access-glftc\") pod \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") "
Nov 01 09:00:04.716707 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.716688    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-metrics-certs\") pod \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") "
Nov 01 09:00:04.717077 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.717059    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls\") pod \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\" (UID: \"8a68e294-5862-46bd-a1f9-c41ae87e3c94\") "
Nov 01 09:00:04.718373 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.715959    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a68e294-5862-46bd-a1f9-c41ae87e3c94-openshift-service-ca" pod "8a68e294-5862-46bd-a1f9-c41ae87e3c94" (UID: "8a68e294-5862-46bd-a1f9-c41ae87e3c94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:04.731469 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.731432    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls" pod "8a68e294-5862-46bd-a1f9-c41ae87e3c94" (UID: "8a68e294-5862-46bd-a1f9-c41ae87e3c94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:00:04.731688 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.731659    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a68e294-5862-46bd-a1f9-c41ae87e3c94-kube-api-access-glftc" pod "8a68e294-5862-46bd-a1f9-c41ae87e3c94" (UID: "8a68e294-5862-46bd-a1f9-c41ae87e3c94"). InnerVolumeSpecName "kube-api-access-glftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:04.738340 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.738124    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-metrics-certs" pod "8a68e294-5862-46bd-a1f9-c41ae87e3c94" (UID: "8a68e294-5862-46bd-a1f9-c41ae87e3c94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:00:04.819628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.819550    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68e294-5862-46bd-a1f9-c41ae87e3c94-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:04.819628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.819579    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glftc\" (UniqueName: \"kubernetes.io/projected/8a68e294-5862-46bd-a1f9-c41ae87e3c94-kube-api-access-glftc\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:04.819628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.819590    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:04.819628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:04.819600    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a68e294-5862-46bd-a1f9-c41ae87e3c94-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:05.453448 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:05.453415    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"]
Nov 01 09:00:05.453861 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:05.453616    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk" podUID="a22f2095-450d-4762-b582-5c12007fa190" containerName="maas-api" containerID="cri-o://fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314" gracePeriod=30
Nov 01 09:00:05.537183 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:05.537145    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7" event={"ID":"8a68e294-5862-46bd-a1f9-c41ae87e3c94","Type":"ContainerDied","Data":"8409b6a9415a6930a375d7b43446b285a22c76f853029a7fd26d043f72bd1dda"}
Nov 01 09:00:05.537404 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:05.537229    4931 scope.go:122] "RemoveContainer" containerID="6fbcbb7bfa7cdb443474416df7811a51ac8f5997a66e7829038a6dbcc5433ab6"
Nov 01 09:00:05.537404 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:05.537157    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"
Nov 01 09:00:05.700049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:05.699392    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"]
Nov 01 09:00:05.744102 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:05.743415    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8701a0-c787ddc7-n9lm7"]
Nov 01 09:00:05.859527 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:05.859444    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 09:00:06.048841 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.048746    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22f2095-450d-4762-b582-5c12007fa190-kube-api-access-gj965\" (UniqueName: \"kubernetes.io/projected/a22f2095-450d-4762-b582-5c12007fa190-kube-api-access-gj965\") pod \"a22f2095-450d-4762-b582-5c12007fa190\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") "
Nov 01 09:00:06.048841 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.048822    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22f2095-450d-4762-b582-5c12007fa190-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22f2095-450d-4762-b582-5c12007fa190-openshift-service-ca\") pod \"a22f2095-450d-4762-b582-5c12007fa190\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") "
Nov 01 09:00:06.049139 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.048862    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls\") pod \"a22f2095-450d-4762-b582-5c12007fa190\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") "
Nov 01 09:00:06.049139 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.048906    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-metrics-certs\") pod \"a22f2095-450d-4762-b582-5c12007fa190\" (UID: \"a22f2095-450d-4762-b582-5c12007fa190\") "
Nov 01 09:00:06.049711 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.049678    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22f2095-450d-4762-b582-5c12007fa190-openshift-service-ca" pod "a22f2095-450d-4762-b582-5c12007fa190" (UID: "a22f2095-450d-4762-b582-5c12007fa190"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:06.049861 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.049836    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22f2095-450d-4762-b582-5c12007fa190-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:06.053909 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.053697    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-metrics-certs" pod "a22f2095-450d-4762-b582-5c12007fa190" (UID: "a22f2095-450d-4762-b582-5c12007fa190"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:00:06.053909 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.053760    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a68e294-5862-46bd-a1f9-c41ae87e3c94" path="/var/lib/kubelet/pods/8a68e294-5862-46bd-a1f9-c41ae87e3c94/volumes"
Nov 01 09:00:06.054180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.053956    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls" pod "a22f2095-450d-4762-b582-5c12007fa190" (UID: "a22f2095-450d-4762-b582-5c12007fa190"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:00:06.054180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.054119    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22f2095-450d-4762-b582-5c12007fa190-kube-api-access-gj965" pod "a22f2095-450d-4762-b582-5c12007fa190" (UID: "a22f2095-450d-4762-b582-5c12007fa190"). InnerVolumeSpecName "kube-api-access-gj965". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:06.151574 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.151544    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj965\" (UniqueName: \"kubernetes.io/projected/a22f2095-450d-4762-b582-5c12007fa190-kube-api-access-gj965\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:06.151574 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.151571    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:06.151840 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.151584    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22f2095-450d-4762-b582-5c12007fa190-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:06.560558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.560521    4931 generic.go:356] "Generic (PLEG): container finished" podID="a22f2095-450d-4762-b582-5c12007fa190" containerID="fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314" exitCode=0
Nov 01 09:00:06.574354 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.574330    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"
Nov 01 09:00:06.578330 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.578270    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk" event={"ID":"a22f2095-450d-4762-b582-5c12007fa190","Type":"ContainerDied","Data":"fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314"}
Nov 01 09:00:06.578679 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.578641    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk" event={"ID":"a22f2095-450d-4762-b582-5c12007fa190","Type":"ContainerDied","Data":"1d0442b867b7b11a2e8dc492468d36239cb12c8444c1d0a4ff8512a823edcfc2"}
Nov 01 09:00:06.578837 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.578824    4931 scope.go:122] "RemoveContainer" containerID="fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314"
Nov 01 09:00:06.600531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.600275    4931 scope.go:122] "RemoveContainer" containerID="fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314"
Nov 01 09:00:06.601105 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:00:06.601010    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314\": container with ID starting with fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314 not found: ID does not exist" containerID="fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314"
Nov 01 09:00:06.601105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.601080    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314"} err="failed to get container status \"fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314\": rpc error: code = NotFound desc = could not find container \"fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314\": container with ID starting with fd5152937cce8507efd0591fce3d0209b73eefbc4d8f82026f7ca3e489b7b314 not found: ID does not exist"
Nov 01 09:00:06.738742 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.738708    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"]
Nov 01 09:00:06.785746 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:06.785705    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5fad29-6ddb9fd558-jvzkk"]
Nov 01 09:00:07.055237 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:07.055203    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"]
Nov 01 09:00:07.055478 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:07.055442    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy" containerID="cri-o://c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd" gracePeriod=30
Nov 01 09:00:08.058144 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:08.056087    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22f2095-450d-4762-b582-5c12007fa190" path="/var/lib/kubelet/pods/a22f2095-450d-4762-b582-5c12007fa190/volumes"
Nov 01 09:00:11.350832 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:11.350803    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"]
Nov 01 09:00:11.351239 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:11.351057    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy" containerID="cri-o://9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57" gracePeriod=30
Nov 01 09:00:12.354443 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.354418    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 09:00:12.476955 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.476911    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-podinfo\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477124 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.476976    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-token\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-token\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477124 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477054    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-data\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477123    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-credential-socket\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477193    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-certs\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477343 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477290    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b095b7-1bea-4711-be17-a9ed5d7907f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b095b7-1bea-4711-be17-a9ed5d7907f1-istiod-ca-cert\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477400 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477342    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-envoy\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477402    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-data" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.477521 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477446    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-credential-socket" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.477521 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477465    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-kube-api-access-phtwh\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-kube-api-access-phtwh\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477521 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477516    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-socket\") pod \"05b095b7-1bea-4711-be17-a9ed5d7907f1\" (UID: \"05b095b7-1bea-4711-be17-a9ed5d7907f1\") "
Nov 01 09:00:12.477684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477652    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-certs" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.477804 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.477785    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b095b7-1bea-4711-be17-a9ed5d7907f1-istiod-ca-cert" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:12.478305 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.478275    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-socket" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.478544 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.478452    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.478629 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.478511    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.478629 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.478589    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.478629 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.478604    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b095b7-1bea-4711-be17-a9ed5d7907f1-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.479789 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.479755    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-podinfo" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:12.480096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.479977    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-envoy" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:12.480197 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.480102    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-token" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.480349 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.480327    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-kube-api-access-phtwh" pod "05b095b7-1bea-4711-be17-a9ed5d7907f1" (UID: "05b095b7-1bea-4711-be17-a9ed5d7907f1"). InnerVolumeSpecName "kube-api-access-phtwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:12.583481 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.583386    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.583724 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.583704    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtwh\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-kube-api-access-phtwh\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.583848 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.583834    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b095b7-1bea-4711-be17-a9ed5d7907f1-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.583945 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.583933    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.584057 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.584036    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b095b7-1bea-4711-be17-a9ed5d7907f1-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:12.631728 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.631682    4931 generic.go:356] "Generic (PLEG): container finished" podID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerID="c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd" exitCode=0
Nov 01 09:00:12.631912 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.631835    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" event={"ID":"05b095b7-1bea-4711-be17-a9ed5d7907f1","Type":"ContainerDied","Data":"c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd"}
Nov 01 09:00:12.631912 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.631888    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9" event={"ID":"05b095b7-1bea-4711-be17-a9ed5d7907f1","Type":"ContainerDied","Data":"291f3ad04b73b85d60f73678848cb333964510b81d3023f636bfcc2311bf2f23"}
Nov 01 09:00:12.632071 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.631916    4931 scope.go:122] "RemoveContainer" containerID="c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd"
Nov 01 09:00:12.634736 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.633729    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"
Nov 01 09:00:12.655415 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.655394    4931 scope.go:122] "RemoveContainer" containerID="c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd"
Nov 01 09:00:12.655981 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:00:12.655949    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd\": container with ID starting with c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd not found: ID does not exist" containerID="c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd"
Nov 01 09:00:12.656106 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.655994    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd"} err="failed to get container status \"c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd\": rpc error: code = NotFound desc = could not find container \"c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd\": container with ID starting with c4f3f88ca0921afb848f1ea32ccdd5580787b88081d642729af06973eb181dbd not found: ID does not exist"
Nov 01 09:00:12.804582 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.804547    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"]
Nov 01 09:00:12.809494 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:12.809455    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-mlxb9"]
Nov 01 09:00:14.050670 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:14.050638    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" path="/var/lib/kubelet/pods/05b095b7-1bea-4711-be17-a9ed5d7907f1/volumes"
Nov 01 09:00:16.638787 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.638696    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 09:00:16.670298 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.670268    4931 generic.go:356] "Generic (PLEG): container finished" podID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerID="9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57" exitCode=0
Nov 01 09:00:16.670458 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.670313    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"
Nov 01 09:00:16.670458 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.670340    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" event={"ID":"02f41188-8e9e-4007-a9d5-43c775e2e3a9","Type":"ContainerDied","Data":"9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57"}
Nov 01 09:00:16.670458 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.670393    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5" event={"ID":"02f41188-8e9e-4007-a9d5-43c775e2e3a9","Type":"ContainerDied","Data":"3abf2130d3e7dd22f56ae27cbaa36c9489f4b09aa11af152c32c2ab67baba38e"}
Nov 01 09:00:16.670458 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.670414    4931 scope.go:122] "RemoveContainer" containerID="9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57"
Nov 01 09:00:16.680464 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.680441    4931 scope.go:122] "RemoveContainer" containerID="9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57"
Nov 01 09:00:16.680759 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:00:16.680736    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57\": container with ID starting with 9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57 not found: ID does not exist" containerID="9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57"
Nov 01 09:00:16.680810 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.680770    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57"} err="failed to get container status \"9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57\": rpc error: code = NotFound desc = could not find container \"9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57\": container with ID starting with 9efa3de717a25d30bd930a950d18267e315077d3ba99aa67758d8185ab058c57 not found: ID does not exist"
Nov 01 09:00:16.752680 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.752634    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-credential-socket\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.752930 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.752688    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-data\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.752930 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.752753    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-certs\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.752930 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.752796    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istiod-ca-cert\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.752930 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.752908    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-kube-api-access-n9mlh\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-kube-api-access-n9mlh\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.753231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.752946    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-credential-socket" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.753231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.752977    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-podinfo\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.753231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.753073    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-data" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.753231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.753225    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-token\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-token\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.753444 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.753313    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-socket\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.753444 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.753339    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istiod-ca-cert" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:16.753444 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.753368    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-envoy\") pod \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\" (UID: \"02f41188-8e9e-4007-a9d5-43c775e2e3a9\") "
Nov 01 09:00:16.753598 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.753554    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-certs" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.753925 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.753894    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-socket" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.754070 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.754017    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.754144 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.754072    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.754144 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.754087    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.754144 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.754101    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.754144 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.754115    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.756464 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.756437    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-envoy" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:16.756628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.756484    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-kube-api-access-n9mlh" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "kube-api-access-n9mlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:16.756628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.756542    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-token" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:16.756864 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.756842    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-podinfo" pod "02f41188-8e9e-4007-a9d5-43c775e2e3a9" (UID: "02f41188-8e9e-4007-a9d5-43c775e2e3a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:16.855653 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.855616    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.855653 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.855655    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.855851 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.855671    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f41188-8e9e-4007-a9d5-43c775e2e3a9-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:16.855851 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:16.855685    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mlh\" (UniqueName: \"kubernetes.io/projected/02f41188-8e9e-4007-a9d5-43c775e2e3a9-kube-api-access-n9mlh\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:17.095165 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:17.095125    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"]
Nov 01 09:00:17.098505 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:17.098477    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-g9zl5"]
Nov 01 09:00:18.054582 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:18.053906    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" path="/var/lib/kubelet/pods/02f41188-8e9e-4007-a9d5-43c775e2e3a9/volumes"
Nov 01 09:00:29.525474 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:29.525439    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"]
Nov 01 09:00:29.525891 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:29.525643    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy" containerID="cri-o://1561835b74635c859f59418d74d6220aa17bd5fe400f89427a69fe53b69def30" gracePeriod=30
Nov 01 09:00:31.023467 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:00:31.023432    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:00:34.820763 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:34.820728    4931 generic.go:356] "Generic (PLEG): container finished" podID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerID="1561835b74635c859f59418d74d6220aa17bd5fe400f89427a69fe53b69def30" exitCode=0
Nov 01 09:00:34.821310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:34.820804    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" event={"ID":"172f6137-4e24-4ad6-b76f-e9c236db3ece","Type":"ContainerDied","Data":"1561835b74635c859f59418d74d6220aa17bd5fe400f89427a69fe53b69def30"}
Nov 01 09:00:34.846189 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:34.846158    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019441    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-credential-socket\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019496    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-data\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019569    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-podinfo\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019639    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-socket\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019666    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-kube-api-access-shlhl\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-kube-api-access-shlhl\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019760    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-envoy\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019786    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-certs\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019876    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-token\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-token\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.029292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.019912    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172f6137-4e24-4ad6-b76f-e9c236db3ece-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172f6137-4e24-4ad6-b76f-e9c236db3ece-istiod-ca-cert\") pod \"172f6137-4e24-4ad6-b76f-e9c236db3ece\" (UID: \"172f6137-4e24-4ad6-b76f-e9c236db3ece\") "
Nov 01 09:00:35.030187 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.030151    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172f6137-4e24-4ad6-b76f-e9c236db3ece-istiod-ca-cert" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:00:35.041085 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.040650    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-data" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.041363 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.041338    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-credential-socket" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.061049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.054280    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-certs" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.061049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.054678    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-socket" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.064463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.064379    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-kube-api-access-shlhl" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "kube-api-access-shlhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:35.067141 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.067086    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-podinfo" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:00:35.070959 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.070916    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-envoy" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:00:35.071577 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.071544    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-token" pod "172f6137-4e24-4ad6-b76f-e9c236db3ece" (UID: "172f6137-4e24-4ad6-b76f-e9c236db3ece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:00:35.145981 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.145918    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.145981 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.145965    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.145981 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.145979    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.146310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.145992    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.146310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.146005    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shlhl\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-kube-api-access-shlhl\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.146310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.146017    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.146310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.146048    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172f6137-4e24-4ad6-b76f-e9c236db3ece-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.146310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.146060    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172f6137-4e24-4ad6-b76f-e9c236db3ece-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.146310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.146072    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172f6137-4e24-4ad6-b76f-e9c236db3ece-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:00:35.830643 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.830562    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"
Nov 01 09:00:35.830643 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.830579    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc" event={"ID":"172f6137-4e24-4ad6-b76f-e9c236db3ece","Type":"ContainerDied","Data":"8e542a14ce5e77d2e92a99b1201b18fc1c8a0341a2fc3b6c4ddf97660d8afeeb"}
Nov 01 09:00:35.830643 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:35.830642    4931 scope.go:122] "RemoveContainer" containerID="1561835b74635c859f59418d74d6220aa17bd5fe400f89427a69fe53b69def30"
Nov 01 09:00:36.022077 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:36.022040    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"]
Nov 01 09:00:36.069357 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:36.069319    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rrpnc"]
Nov 01 09:00:38.050373 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:00:38.050328    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" path="/var/lib/kubelet/pods/172f6137-4e24-4ad6-b76f-e9c236db3ece/volumes"
Nov 01 09:01:01.029858 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:01:01.029796    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:01:31.023092 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:01:31.023015    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:01:45.734530 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:01:45.734491    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"]
Nov 01 09:02:00.756957 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:00.756917    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6" podUID="5405ff20-f30b-4c5c-818f-b5ab5a06a940" containerName="main" containerID="cri-o://0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a" gracePeriod=45
Nov 01 09:02:01.033211 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:02:01.033161    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:02:01.062100 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.062073    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 09:02:01.161072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.160976    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5405ff20-f30b-4c5c-818f-b5ab5a06a940-kube-api-access-dwdfm\" (UniqueName: \"kubernetes.io/projected/5405ff20-f30b-4c5c-818f-b5ab5a06a940-kube-api-access-dwdfm\") pod \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") "
Nov 01 09:02:01.161072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161047    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tls-certs\" (UniqueName: \"kubernetes.io/secret/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tls-certs\") pod \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") "
Nov 01 09:02:01.161374 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161139    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-model-cache\") pod \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") "
Nov 01 09:02:01.161374 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161194    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-home\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-home\") pod \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") "
Nov 01 09:02:01.161374 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161221    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tmp-dir\") pod \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") "
Nov 01 09:02:01.161615 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161439    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-dshm\") pod \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\" (UID: \"5405ff20-f30b-4c5c-818f-b5ab5a06a940\") "
Nov 01 09:02:01.161615 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161449    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-home" pod "5405ff20-f30b-4c5c-818f-b5ab5a06a940" (UID: "5405ff20-f30b-4c5c-818f-b5ab5a06a940"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:01.161615 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161436    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-model-cache" pod "5405ff20-f30b-4c5c-818f-b5ab5a06a940" (UID: "5405ff20-f30b-4c5c-818f-b5ab5a06a940"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:01.161796 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161707    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tmp-dir" pod "5405ff20-f30b-4c5c-818f-b5ab5a06a940" (UID: "5405ff20-f30b-4c5c-818f-b5ab5a06a940"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:01.161999 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161943    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:01.161999 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161969    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:01.161999 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.161986    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:01.163662 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.163634    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5405ff20-f30b-4c5c-818f-b5ab5a06a940-kube-api-access-dwdfm" pod "5405ff20-f30b-4c5c-818f-b5ab5a06a940" (UID: "5405ff20-f30b-4c5c-818f-b5ab5a06a940"). InnerVolumeSpecName "kube-api-access-dwdfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:01.163790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.163727    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-dshm" pod "5405ff20-f30b-4c5c-818f-b5ab5a06a940" (UID: "5405ff20-f30b-4c5c-818f-b5ab5a06a940"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:02:01.164110 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.164085    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tls-certs" pod "5405ff20-f30b-4c5c-818f-b5ab5a06a940" (UID: "5405ff20-f30b-4c5c-818f-b5ab5a06a940"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:02:01.262746 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.262711    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5405ff20-f30b-4c5c-818f-b5ab5a06a940-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:01.262746 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.262739    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwdfm\" (UniqueName: \"kubernetes.io/projected/5405ff20-f30b-4c5c-818f-b5ab5a06a940-kube-api-access-dwdfm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:01.262746 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.262751    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5405ff20-f30b-4c5c-818f-b5ab5a06a940-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:01.546109 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.546049    4931 generic.go:356] "Generic (PLEG): container finished" podID="5405ff20-f30b-4c5c-818f-b5ab5a06a940" containerID="0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a" exitCode=0
Nov 01 09:02:01.546109 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.546106    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"
Nov 01 09:02:01.546365 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.546128    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6" event={"ID":"5405ff20-f30b-4c5c-818f-b5ab5a06a940","Type":"ContainerDied","Data":"0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a"}
Nov 01 09:02:01.546365 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.546179    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6" event={"ID":"5405ff20-f30b-4c5c-818f-b5ab5a06a940","Type":"ContainerDied","Data":"fb65c96488d01c027eaebbd92ef45be3e1bd3b96b127964ebdec48735239f9f2"}
Nov 01 09:02:01.546365 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.546201    4931 scope.go:122] "RemoveContainer" containerID="0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a"
Nov 01 09:02:01.557206 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.557184    4931 scope.go:122] "RemoveContainer" containerID="0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a"
Nov 01 09:02:01.557532 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:02:01.557509    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a\": container with ID starting with 0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a not found: ID does not exist" containerID="0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a"
Nov 01 09:02:01.557615 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.557539    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a"} err="failed to get container status \"0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a\": rpc error: code = NotFound desc = could not find container \"0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a\": container with ID starting with 0444f7303c481dffca0994b85fde336100d585a190265d8c7d221cfc414f220a not found: ID does not exist"
Nov 01 09:02:01.706156 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.706119    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"]
Nov 01 09:02:01.753835 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:01.753798    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d31acd45/e2e-lifecycle-model-d31acd45-kserve-68769dcbfc-mfvh6"]
Nov 01 09:02:02.049006 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:02.048976    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5405ff20-f30b-4c5c-818f-b5ab5a06a940" path="/var/lib/kubelet/pods/5405ff20-f30b-4c5c-818f-b5ab5a06a940/volumes"
Nov 01 09:02:10.327717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:10.327684    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:02:10.337208 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:10.337137    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:02:10.361820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:10.361792    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:02:10.366823 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:10.366793    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:02:10.460367 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:10.460332    4931 scope.go:122] "RemoveContainer" containerID="b554d40b1774e92d2848070402fea4dacbeb8996e26a60561a8831627658bcba"
Nov 01 09:02:31.024329 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:02:31.024286    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:02:42.385540 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.385496    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"]
Nov 01 09:02:42.386576 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.385703    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b" podUID="27669d73-86a4-4f64-8582-23ac0072cd81" containerName="maas-api" containerID="cri-o://4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe" gracePeriod=30
Nov 01 09:02:42.778583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.778554    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 09:02:42.881476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.880273    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-metrics-certs\") pod \"27669d73-86a4-4f64-8582-23ac0072cd81\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") "
Nov 01 09:02:42.881476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.880417    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27669d73-86a4-4f64-8582-23ac0072cd81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27669d73-86a4-4f64-8582-23ac0072cd81-openshift-service-ca\") pod \"27669d73-86a4-4f64-8582-23ac0072cd81\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") "
Nov 01 09:02:42.881476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.880446    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls\") pod \"27669d73-86a4-4f64-8582-23ac0072cd81\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") "
Nov 01 09:02:42.881476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.880580    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27669d73-86a4-4f64-8582-23ac0072cd81-kube-api-access-ndwrh\" (UniqueName: \"kubernetes.io/projected/27669d73-86a4-4f64-8582-23ac0072cd81-kube-api-access-ndwrh\") pod \"27669d73-86a4-4f64-8582-23ac0072cd81\" (UID: \"27669d73-86a4-4f64-8582-23ac0072cd81\") "
Nov 01 09:02:42.882315 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.882275    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27669d73-86a4-4f64-8582-23ac0072cd81-openshift-service-ca" pod "27669d73-86a4-4f64-8582-23ac0072cd81" (UID: "27669d73-86a4-4f64-8582-23ac0072cd81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:02:42.891897 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.890550    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-metrics-certs" pod "27669d73-86a4-4f64-8582-23ac0072cd81" (UID: "27669d73-86a4-4f64-8582-23ac0072cd81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:02:42.933342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.925634    4931 generic.go:356] "Generic (PLEG): container finished" podID="27669d73-86a4-4f64-8582-23ac0072cd81" containerID="4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe" exitCode=0
Nov 01 09:02:42.933342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.925848    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b" event={"ID":"27669d73-86a4-4f64-8582-23ac0072cd81","Type":"ContainerDied","Data":"4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe"}
Nov 01 09:02:42.933342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.925890    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b" event={"ID":"27669d73-86a4-4f64-8582-23ac0072cd81","Type":"ContainerDied","Data":"717eed4967c6d81129f5b3ed4b72a5d55c090267aeddd5900ff54af400f058f2"}
Nov 01 09:02:42.933342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.925911    4931 scope.go:122] "RemoveContainer" containerID="4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe"
Nov 01 09:02:42.933342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.926085    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"
Nov 01 09:02:42.933342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.929349    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27669d73-86a4-4f64-8582-23ac0072cd81-kube-api-access-ndwrh" pod "27669d73-86a4-4f64-8582-23ac0072cd81" (UID: "27669d73-86a4-4f64-8582-23ac0072cd81"). InnerVolumeSpecName "kube-api-access-ndwrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:02:42.951946 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.951880    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls" pod "27669d73-86a4-4f64-8582-23ac0072cd81" (UID: "27669d73-86a4-4f64-8582-23ac0072cd81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:02:42.957581 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.957332    4931 scope.go:122] "RemoveContainer" containerID="4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe"
Nov 01 09:02:42.958305 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:02:42.958189    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe\": container with ID starting with 4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe not found: ID does not exist" containerID="4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe"
Nov 01 09:02:42.958305 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.958235    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe"} err="failed to get container status \"4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe\": rpc error: code = NotFound desc = could not find container \"4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe\": container with ID starting with 4a90cdeb00c71f8f3b5adcedafa469a0ff5fb83e4723fba07e99ba30f6f03dfe not found: ID does not exist"
Nov 01 09:02:42.982594 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.982560    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27669d73-86a4-4f64-8582-23ac0072cd81-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:42.982594 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.982598    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:42.982862 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.982617    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndwrh\" (UniqueName: \"kubernetes.io/projected/27669d73-86a4-4f64-8582-23ac0072cd81-kube-api-access-ndwrh\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:42.982862 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:42.982633    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27669d73-86a4-4f64-8582-23ac0072cd81-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:02:43.343243 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:43.343209    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"]
Nov 01 09:02:43.418054 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:43.417966    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03832c-6dc4b78fdb-m5f4b"]
Nov 01 09:02:44.052082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:02:44.052043    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27669d73-86a4-4f64-8582-23ac0072cd81" path="/var/lib/kubelet/pods/27669d73-86a4-4f64-8582-23ac0072cd81/volumes"
Nov 01 09:03:01.029590 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:03:01.029505    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:03:31.028646 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:03:31.028608    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:04:01.063282 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:04:01.063238    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:04:31.024286 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:04:31.024194    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:05:01.023899 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:05:01.023851    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:05:09.328845 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:09.328799    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"]
Nov 01 09:05:09.329612 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:09.329145    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" containerID="cri-o://fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3" gracePeriod=30
Nov 01 09:05:11.096180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:11.096138    4931 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:05:11.096575 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:11.096190    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:05:14.596266 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.596231    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 09:05:14.681233 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681201    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-token\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-token\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681233 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681244    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istiod-ca-cert\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681467 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681265    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-data\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681467 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681292    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-certs\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681467 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681322    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-envoy\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681467 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681368    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-kube-api-access-djg4b\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-kube-api-access-djg4b\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681467 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681436    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-credential-socket\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681470    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-podinfo\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681590    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-socket\") pod \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\" (UID: \"01d8333e-26a1-46ca-9ad4-2d6dc20d441b\") "
Nov 01 09:05:14.681729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681615    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-data" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:14.681729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681672    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-certs" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:14.681729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681715    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istiod-ca-cert" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:14.681991 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.681966    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-socket" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:14.682116 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.682081    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-credential-socket" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:14.682313 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.682273    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:14.682313 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.682299    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:14.682442 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.682315    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:14.682442 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.682328    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:14.682442 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.682341    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:14.683858 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.683832    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-podinfo" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:14.683974 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.683877    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-envoy" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:14.684215 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.684195    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-token" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:14.684323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.684286    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-kube-api-access-djg4b" pod "01d8333e-26a1-46ca-9ad4-2d6dc20d441b" (UID: "01d8333e-26a1-46ca-9ad4-2d6dc20d441b"). InnerVolumeSpecName "kube-api-access-djg4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:14.784107 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.784071    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:14.784107 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.784106    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:14.784372 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.784120    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djg4b\" (UniqueName: \"kubernetes.io/projected/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-kube-api-access-djg4b\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:14.784372 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:14.784135    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d8333e-26a1-46ca-9ad4-2d6dc20d441b-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:15.278275 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.278239    4931 generic.go:356] "Generic (PLEG): container finished" podID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerID="fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3" exitCode=0
Nov 01 09:05:15.278446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.278283    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"
Nov 01 09:05:15.278446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.278319    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" event={"ID":"01d8333e-26a1-46ca-9ad4-2d6dc20d441b","Type":"ContainerDied","Data":"fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3"}
Nov 01 09:05:15.278446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.278379    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d" event={"ID":"01d8333e-26a1-46ca-9ad4-2d6dc20d441b","Type":"ContainerDied","Data":"67d9d53f57441d72970740c7f1a4d322e7acac2b4831f6094648462851ff9109"}
Nov 01 09:05:15.278446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.278405    4931 scope.go:122] "RemoveContainer" containerID="fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3"
Nov 01 09:05:15.289845 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.289819    4931 scope.go:122] "RemoveContainer" containerID="fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3"
Nov 01 09:05:15.290239 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:05:15.290217    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3\": container with ID starting with fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3 not found: ID does not exist" containerID="fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3"
Nov 01 09:05:15.290352 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.290248    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3"} err="failed to get container status \"fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3\": rpc error: code = NotFound desc = could not find container \"fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3\": container with ID starting with fe595c6efa193f0728268ae99c57822de302a573d00aaf72f0de94aeee5125d3 not found: ID does not exist"
Nov 01 09:05:15.438180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.438145    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"]
Nov 01 09:05:15.442556 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:15.442530    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-5lz5d"]
Nov 01 09:05:16.063567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:16.063501    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" path="/var/lib/kubelet/pods/01d8333e-26a1-46ca-9ad4-2d6dc20d441b/volumes"
Nov 01 09:05:29.276721 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:29.276677    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"]
Nov 01 09:05:29.277689 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:29.276998    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" containerID="cri-o://6fa5fb15ac9114d5a8a5785714f18a43d0056f3d80792720add07405dba0baf7" gracePeriod=30
Nov 01 09:05:29.321331 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:29.321279    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"]
Nov 01 09:05:29.321887 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:29.321853    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy" containerID="cri-o://490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78" gracePeriod=30
Nov 01 09:05:31.026102 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:05:31.026069    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:05:31.822867 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:31.822770    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"]
Nov 01 09:05:31.823221 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:31.823037    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" containerID="cri-o://1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407" gracePeriod=30
Nov 01 09:05:31.870943 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:31.870911    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"]
Nov 01 09:05:31.871142 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:31.871117    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy" containerID="cri-o://9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a" gracePeriod=30
Nov 01 09:05:34.434752 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.434661    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fspks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 09:05:34.434752 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.434742    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 09:05:34.437012 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.436987    4931 generic.go:356] "Generic (PLEG): container finished" podID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerID="6fa5fb15ac9114d5a8a5785714f18a43d0056f3d80792720add07405dba0baf7" exitCode=0
Nov 01 09:05:34.437138 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.437069    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" event={"ID":"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8","Type":"ContainerDied","Data":"6fa5fb15ac9114d5a8a5785714f18a43d0056f3d80792720add07405dba0baf7"}
Nov 01 09:05:34.629301 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.629276    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 09:05:34.669632 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.669606    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 09:05:34.716262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716229    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-data\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716421 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716397    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-credential-socket\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716462 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716435    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-certs\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716496 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716483    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-token\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-token\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716534 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716510    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-envoy\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716581 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716535    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-data" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.716581 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716560    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-podinfo\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716587    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-kube-api-access-sdr75\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-kube-api-access-sdr75\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716737 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716708    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0edf08ed-02ef-4a05-9abf-dbe60053f296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edf08ed-02ef-4a05-9abf-dbe60053f296-istiod-ca-cert\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716737 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716702    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-credential-socket" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.716840 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716803    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-socket\") pod \"0edf08ed-02ef-4a05-9abf-dbe60053f296\" (UID: \"0edf08ed-02ef-4a05-9abf-dbe60053f296\") "
Nov 01 09:05:34.716840 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.716821    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-certs" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.717333 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.717312    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.717333 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.717333    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.717531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.717342    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.717906 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.717868    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0edf08ed-02ef-4a05-9abf-dbe60053f296-istiod-ca-cert" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:34.717968 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.717943    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-socket" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.719077 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.719055    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-token" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:34.719341 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.719305    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-podinfo" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:34.719444 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.719372    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-envoy" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.719632 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.719615    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-kube-api-access-sdr75" pod "0edf08ed-02ef-4a05-9abf-dbe60053f296" (UID: "0edf08ed-02ef-4a05-9abf-dbe60053f296"). InnerVolumeSpecName "kube-api-access-sdr75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:34.818671 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.818634    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-data\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.818883 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.818679    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-socket\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.818883 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.818732    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-token\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-token\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.818883 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.818790    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istiod-ca-cert\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.818883 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.818864    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-credential-socket\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.819141 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.818981    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-certs\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.819141 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.819016    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-podinfo\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.819141 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.819108    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-kube-api-access-zdnfc\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-kube-api-access-zdnfc\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.819458 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.819146    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-envoy\") pod \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\" (UID: \"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8\") "
Nov 01 09:05:34.819458 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.819374    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-credential-socket" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.819620 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.819465    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-socket" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.819620 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.819590    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-certs" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.820159 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.819905    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-data" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.820159 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820147    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istiod-ca-cert" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:34.820329 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820313    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820334    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820350    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820364    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edf08ed-02ef-4a05-9abf-dbe60053f296-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820380    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdr75\" (UniqueName: \"kubernetes.io/projected/0edf08ed-02ef-4a05-9abf-dbe60053f296-kube-api-access-sdr75\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820394    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820407    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edf08ed-02ef-4a05-9abf-dbe60053f296-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820422    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edf08ed-02ef-4a05-9abf-dbe60053f296-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820435    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820450    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.820931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.820465    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.822092 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.822005    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-envoy" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:34.822331 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.822311    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-kube-api-access-zdnfc" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "kube-api-access-zdnfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:34.823548 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.823518    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-token" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:34.823650 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.823627    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-podinfo" pod "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" (UID: "23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:34.921281 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.921245    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdnfc\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-kube-api-access-zdnfc\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.921281 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.921277    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.921281 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.921287    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:34.921575 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:34.921296    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:35.446420 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.446387    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"
Nov 01 09:05:35.446886 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.446417    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks" event={"ID":"23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8","Type":"ContainerDied","Data":"abf2fb1217da76aceb59db3f0c48d7ec99ae846cfc353a85c952c807b6dab953"}
Nov 01 09:05:35.446886 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.446493    4931 scope.go:122] "RemoveContainer" containerID="6fa5fb15ac9114d5a8a5785714f18a43d0056f3d80792720add07405dba0baf7"
Nov 01 09:05:35.448128 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.448111    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"
Nov 01 09:05:35.448225 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.448128    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" event={"ID":"0edf08ed-02ef-4a05-9abf-dbe60053f296","Type":"ContainerDied","Data":"490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78"}
Nov 01 09:05:35.448225 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.448018    4931 generic.go:356] "Generic (PLEG): container finished" podID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerID="490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78" exitCode=0
Nov 01 09:05:35.448314 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.448285    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl" event={"ID":"0edf08ed-02ef-4a05-9abf-dbe60053f296","Type":"ContainerDied","Data":"553484d89e06a0ddd36dc67f795566c73fd5e5cc17525ff32d16bece76487d3d"}
Nov 01 09:05:35.461096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.460795    4931 scope.go:122] "RemoveContainer" containerID="490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78"
Nov 01 09:05:35.471236 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.471215    4931 scope.go:122] "RemoveContainer" containerID="490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78"
Nov 01 09:05:35.471549 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:05:35.471534    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78\": container with ID starting with 490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78 not found: ID does not exist" containerID="490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78"
Nov 01 09:05:35.471601 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.471561    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78"} err="failed to get container status \"490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78\": rpc error: code = NotFound desc = could not find container \"490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78\": container with ID starting with 490fda25cccc21288f0357fb22f641ca3d72a8597ff4141679bf0bd050d78f78 not found: ID does not exist"
Nov 01 09:05:35.545086 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.545052    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:05:35.545274 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.545127    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:05:35.646311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.646276    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"]
Nov 01 09:05:35.656666 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.656638    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fspks"]
Nov 01 09:05:35.804211 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.804175    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"]
Nov 01 09:05:35.810229 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:35.810204    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-b2jtl"]
Nov 01 09:05:36.049520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:36.049437    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" path="/var/lib/kubelet/pods/0edf08ed-02ef-4a05-9abf-dbe60053f296/volumes"
Nov 01 09:05:36.049899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:36.049886    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" path="/var/lib/kubelet/pods/23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8/volumes"
Nov 01 09:05:37.155578 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.155554    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 09:05:37.201294 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.201269    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 09:05:37.254481 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254450    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istiod-ca-cert\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.254676 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254491    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-credential-socket\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.254676 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254527    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-envoy\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.254676 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254546    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-socket\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.254676 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254593    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-token\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.254676 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254648    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-istio-token\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-istio-token\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.254938 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254688    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-kube-api-access-b8s4f\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-kube-api-access-b8s4f\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.254938 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254757    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-podinfo\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.254938 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254790    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-envoy\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.254938 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254817    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-credential-socket" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.254938 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254836    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-certs\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.254938 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254841    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-socket" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.254938 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.254871    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istiod-ca-cert" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:37.255296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255110    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-certs" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.255296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255115    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d44de48-7523-455d-8788-5ba07f9d6457-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d44de48-7523-455d-8788-5ba07f9d6457-istio-podinfo\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.255296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255161    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-kube-api-access-r5m7r\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-kube-api-access-r5m7r\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.255296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255206    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-certs\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.255296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255254    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-data\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.255532 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255300    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-data\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.255532 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255372    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d44de48-7523-455d-8788-5ba07f9d6457-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d44de48-7523-455d-8788-5ba07f9d6457-istiod-ca-cert\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.255532 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255403    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-socket\") pod \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\" (UID: \"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677\") "
Nov 01 09:05:37.255684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.255611    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-credential-socket\") pod \"8d44de48-7523-455d-8788-5ba07f9d6457\" (UID: \"8d44de48-7523-455d-8788-5ba07f9d6457\") "
Nov 01 09:05:37.256335 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.256313    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.256406 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.256343    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.256406 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.256359    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.256406 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.256373    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.256604 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.256584    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-credential-socket" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.257683 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.257655    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-envoy" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.257809 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.257751    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-podinfo" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:37.257871 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.257859    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-data" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.257966 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.257939    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d44de48-7523-455d-8788-5ba07f9d6457-istio-podinfo" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:05:37.258146 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.258121    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-certs" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.258146 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.258131    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-kube-api-access-b8s4f" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "kube-api-access-b8s4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:37.258540 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.258312    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d44de48-7523-455d-8788-5ba07f9d6457-istiod-ca-cert" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:05:37.258540 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.258388    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-socket" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.258540 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.258514    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-envoy" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.258779 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.258552    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-data" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:05:37.259257 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.258999    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-istio-token" pod "8d44de48-7523-455d-8788-5ba07f9d6457" (UID: "8d44de48-7523-455d-8788-5ba07f9d6457"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:37.259728 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.259704    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-token" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:37.260264 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.260235    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-kube-api-access-r5m7r" pod "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" (UID: "d3fe99a6-e1a0-4dfb-9062-33d37cb8e677"). InnerVolumeSpecName "kube-api-access-r5m7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:05:37.357372 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357294    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357372 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357323    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357372 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357338    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8s4f\" (UniqueName: \"kubernetes.io/projected/8d44de48-7523-455d-8788-5ba07f9d6457-kube-api-access-b8s4f\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357372 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357352    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357372 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357365    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357379    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d44de48-7523-455d-8788-5ba07f9d6457-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357391    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5m7r\" (UniqueName: \"kubernetes.io/projected/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-kube-api-access-r5m7r\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357408    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357417    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357428    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357441    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d44de48-7523-455d-8788-5ba07f9d6457-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357452    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357463    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.357765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.357476    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d44de48-7523-455d-8788-5ba07f9d6457-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:05:37.473063 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.470894    4931 generic.go:356] "Generic (PLEG): container finished" podID="8d44de48-7523-455d-8788-5ba07f9d6457" containerID="9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a" exitCode=0
Nov 01 09:05:37.473063 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.470959    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"
Nov 01 09:05:37.473063 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.470978    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" event={"ID":"8d44de48-7523-455d-8788-5ba07f9d6457","Type":"ContainerDied","Data":"9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a"}
Nov 01 09:05:37.473063 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.471089    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn" event={"ID":"8d44de48-7523-455d-8788-5ba07f9d6457","Type":"ContainerDied","Data":"cfbfabf29e91d6bca591b63130b59d5de7e21c7bc85d376e7929f80b648f801a"}
Nov 01 09:05:37.473063 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.471122    4931 scope.go:122] "RemoveContainer" containerID="9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a"
Nov 01 09:05:37.473646 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.473620    4931 generic.go:356] "Generic (PLEG): container finished" podID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerID="1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407" exitCode=0
Nov 01 09:05:37.473748 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.473730    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" event={"ID":"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677","Type":"ContainerDied","Data":"1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407"}
Nov 01 09:05:37.473804 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.473777    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp" event={"ID":"d3fe99a6-e1a0-4dfb-9062-33d37cb8e677","Type":"ContainerDied","Data":"e581723563efe60e18b5858f9cba981ca02691786f23f5bb5d6e80c46a639f8d"}
Nov 01 09:05:37.473880 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.473866    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"
Nov 01 09:05:37.499724 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.499685    4931 scope.go:122] "RemoveContainer" containerID="9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a"
Nov 01 09:05:37.500136 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:05:37.500111    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a\": container with ID starting with 9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a not found: ID does not exist" containerID="9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a"
Nov 01 09:05:37.500217 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.500146    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a"} err="failed to get container status \"9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a\": rpc error: code = NotFound desc = could not find container \"9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a\": container with ID starting with 9973a23b270317797271fd8a07c47a8a2be956bb77d412e285c4029acf5c0f0a not found: ID does not exist"
Nov 01 09:05:37.500217 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.500166    4931 scope.go:122] "RemoveContainer" containerID="1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407"
Nov 01 09:05:37.511488 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.511469    4931 scope.go:122] "RemoveContainer" containerID="1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407"
Nov 01 09:05:37.511874 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:05:37.511853    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407\": container with ID starting with 1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407 not found: ID does not exist" containerID="1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407"
Nov 01 09:05:37.511928 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.511886    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407"} err="failed to get container status \"1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407\": rpc error: code = NotFound desc = could not find container \"1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407\": container with ID starting with 1437e6ba624b8971d2ff6d86d18dd5e38555b4400c8a14ef958ee2696bee3407 not found: ID does not exist"
Nov 01 09:05:37.631386 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.631290    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"]
Nov 01 09:05:37.635140 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.635113    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-brnkn"]
Nov 01 09:05:37.739628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.739589    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"]
Nov 01 09:05:37.745883 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:37.745852    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-l6kfp"]
Nov 01 09:05:38.048335 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:38.048261    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" path="/var/lib/kubelet/pods/8d44de48-7523-455d-8788-5ba07f9d6457/volumes"
Nov 01 09:05:38.048760 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:05:38.048747    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" path="/var/lib/kubelet/pods/d3fe99a6-e1a0-4dfb-9062-33d37cb8e677/volumes"
Nov 01 09:06:01.023794 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:06:01.023752    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:06:31.022542 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:06:31.022500    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:07:01.022742 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:07:01.022698    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:07:10.436751 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:10.436723    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:07:10.441104 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:10.441081    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:07:10.457923 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:10.457899    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:07:10.461980 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:10.461958    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:07:13.925142 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925107    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"]
Nov 01 09:07:13.925839 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925817    4931 cpu_manager.go:414] "containerMap: removing container" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925841    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925861    4931 cpu_manager.go:414] "containerMap: removing container" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925867    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925882    4931 cpu_manager.go:414] "containerMap: removing container" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925892    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925906    4931 cpu_manager.go:414] "containerMap: removing container" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925913    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925928    4931 cpu_manager.go:414] "containerMap: removing container" podUID="e23d00cb-4858-40a3-baf4-6e1429ae747b" containerName="maas-api"
Nov 01 09:07:13.925937 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925934    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23d00cb-4858-40a3-baf4-6e1429ae747b" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925963    4931 cpu_manager.go:414] "containerMap: removing container" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925971    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925980    4931 cpu_manager.go:414] "containerMap: removing container" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.925988    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926006    4931 cpu_manager.go:414] "containerMap: removing container" podUID="27669d73-86a4-4f64-8582-23ac0072cd81" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926014    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27669d73-86a4-4f64-8582-23ac0072cd81" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926055    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a22f2095-450d-4762-b582-5c12007fa190" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926063    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22f2095-450d-4762-b582-5c12007fa190" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926086    4931 cpu_manager.go:414] "containerMap: removing container" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926094    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926107    4931 cpu_manager.go:414] "containerMap: removing container" podUID="5405ff20-f30b-4c5c-818f-b5ab5a06a940" containerName="main"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926115    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5405ff20-f30b-4c5c-818f-b5ab5a06a940" containerName="main"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926128    4931 cpu_manager.go:414] "containerMap: removing container" podUID="8a68e294-5862-46bd-a1f9-c41ae87e3c94" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926135    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a68e294-5862-46bd-a1f9-c41ae87e3c94" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926150    4931 cpu_manager.go:414] "containerMap: removing container" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926157    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926310    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d8333e-26a1-46ca-9ad4-2d6dc20d441b" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926328    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fe99a6-e1a0-4dfb-9062-33d37cb8e677" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926338    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="172f6137-4e24-4ad6-b76f-e9c236db3ece" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926348    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23d00cb-4858-40a3-baf4-6e1429ae747b" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926357    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22f2095-450d-4762-b582-5c12007fa190" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926370    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d44de48-7523-455d-8788-5ba07f9d6457" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926379    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5405ff20-f30b-4c5c-818f-b5ab5a06a940" containerName="main"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926390    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f41188-8e9e-4007-a9d5-43c775e2e3a9" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926399    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0edf08ed-02ef-4a05-9abf-dbe60053f296" containerName="istio-proxy"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926407    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a68e294-5862-46bd-a1f9-c41ae87e3c94" containerName="maas-api"
Nov 01 09:07:13.926407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926415    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b095b7-1bea-4711-be17-a9ed5d7907f1" containerName="istio-proxy"
Nov 01 09:07:13.927648 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926424    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a71f55-bb5e-4c2d-a7fd-fb4c4636a2b8" containerName="istio-proxy"
Nov 01 09:07:13.927648 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.926434    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="27669d73-86a4-4f64-8582-23ac0072cd81" containerName="maas-api"
Nov 01 09:07:13.930429 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.930407    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:13.980597 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:13.980566    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c0460d-serving-cert\""
Nov 01 09:07:14.033861 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.033826    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"]
Nov 01 09:07:14.114885 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.114842    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.114885 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.114888    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466367d6-f226-4d59-bbe1-8b5ffc058318-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.115210 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.115188    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rppj\" (UniqueName: \"kubernetes.io/projected/466367d6-f226-4d59-bbe1-8b5ffc058318-kube-api-access-7rppj\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.115251 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.115230    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.216569 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.216478    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.216569 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.216522    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466367d6-f226-4d59-bbe1-8b5ffc058318-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.216815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.216677    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rppj\" (UniqueName: \"kubernetes.io/projected/466367d6-f226-4d59-bbe1-8b5ffc058318-kube-api-access-7rppj\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.216815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.216716    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.217375 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.217348    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466367d6-f226-4d59-bbe1-8b5ffc058318-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.219557 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.219532    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.219742 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.219720    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.276298 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.276259    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rppj\" (UniqueName: \"kubernetes.io/projected/466367d6-f226-4d59-bbe1-8b5ffc058318-kube-api-access-7rppj\") pod \"maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.545714 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.545619    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:14.926389 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.926357    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"]
Nov 01 09:07:14.927074 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:07:14.927002    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466367d6_f226_4d59_bbe1_8b5ffc058318.slice/crio-5e1bf3a59033a4048b870c3427fbf106b34ea3668b397ac30f629a7668deb75c WatchSource:0}: Error finding container 5e1bf3a59033a4048b870c3427fbf106b34ea3668b397ac30f629a7668deb75c: Status 404 returned error can't find the container with id 5e1bf3a59033a4048b870c3427fbf106b34ea3668b397ac30f629a7668deb75c
Nov 01 09:07:14.928433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:14.928412    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:07:15.211482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:15.211391    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw" event={"ID":"466367d6-f226-4d59-bbe1-8b5ffc058318","Type":"ContainerStarted","Data":"5e1bf3a59033a4048b870c3427fbf106b34ea3668b397ac30f629a7668deb75c"}
Nov 01 09:07:16.220193 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:16.220154    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw" event={"ID":"466367d6-f226-4d59-bbe1-8b5ffc058318","Type":"ContainerStarted","Data":"21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8"}
Nov 01 09:07:16.220618 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:16.220376    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:16.334916 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:16.334862    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw" podStartSLOduration=2.477335679 podStartE2EDuration="3.334846423s" totalImagesPullingTime="857.510744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:16.333714037 +0000 UTC m=+2106.448587006" watchObservedRunningTime="2026-11-01 09:07:16.334846423 +0000 UTC m=+2106.449719390"
Nov 01 09:07:22.231944 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:22.231855    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:07:31.023531 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:07:31.023490    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:07:53.554640 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.554604    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"]
Nov 01 09:07:53.558868 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.558848    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.658528 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.658493    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.658722 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.658561    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.658722 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.658639    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-metrics-certs\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.658722 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.658678    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4j6t\" (UniqueName: \"kubernetes.io/projected/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-kube-api-access-c4j6t\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.658861 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.658734    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4be7cc-serving-cert\""
Nov 01 09:07:53.661970 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.661950    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"]
Nov 01 09:07:53.760058 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.760010    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.760260 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.760093    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.760260 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:07:53.760136    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-serving-cert: secret "maas-api-e2e-ipp-b-4be7cc-serving-cert" not found
Nov 01 09:07:53.760260 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:07:53.760213    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls podName:e1cdc679-f48f-4f89-bec2-1048ab60fbcb nodeName:}" failed. No retries permitted until 2026-11-01 09:07:54.260191735 +0000 UTC m=+2144.375064683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls") pod "maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w" (UID: "e1cdc679-f48f-4f89-bec2-1048ab60fbcb") : secret "maas-api-e2e-ipp-b-4be7cc-serving-cert" not found
Nov 01 09:07:53.760449 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.760137    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-metrics-certs\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.760519 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.760458    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4j6t\" (UniqueName: \"kubernetes.io/projected/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-kube-api-access-c4j6t\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.760945 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.760920    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.763019 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.762997    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-metrics-certs\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:53.821323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:53.821252    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4j6t\" (UniqueName: \"kubernetes.io/projected/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-kube-api-access-c4j6t\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:54.270454 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:54.270415    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:54.274186 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:54.274153    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:54.476602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:54.476320    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:55.027246 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:55.027217    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"]
Nov 01 09:07:55.027745 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:07:55.027723    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cdc679_f48f_4f89_bec2_1048ab60fbcb.slice/crio-340ba2ea73ac304304024caba08069123e6e046530f508a80787b30108ef4d6b WatchSource:0}: Error finding container 340ba2ea73ac304304024caba08069123e6e046530f508a80787b30108ef4d6b: Status 404 returned error can't find the container with id 340ba2ea73ac304304024caba08069123e6e046530f508a80787b30108ef4d6b
Nov 01 09:07:55.626893 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:55.626857    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w" event={"ID":"e1cdc679-f48f-4f89-bec2-1048ab60fbcb","Type":"ContainerStarted","Data":"340ba2ea73ac304304024caba08069123e6e046530f508a80787b30108ef4d6b"}
Nov 01 09:07:56.635808 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:56.635760    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w" event={"ID":"e1cdc679-f48f-4f89-bec2-1048ab60fbcb","Type":"ContainerStarted","Data":"cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6"}
Nov 01 09:07:56.636257 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:56.635835    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:07:56.749690 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:07:56.749635    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w" podStartSLOduration=2.886449355 podStartE2EDuration="3.74962164s" totalImagesPullingTime="863.172285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:56.748098662 +0000 UTC m=+2146.862971630" watchObservedRunningTime="2026-11-01 09:07:56.74962164 +0000 UTC m=+2146.864494609"
Nov 01 09:08:01.024443 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:08:01.024399    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:08:02.648832 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:08:02.648803    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:08:31.024289 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:08:31.024228    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:09:00.296344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:00.296247    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"]
Nov 01 09:09:00.297516 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:00.296510    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy" containerID="cri-o://c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e" gracePeriod=30
Nov 01 09:09:01.023512 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:09:01.023476    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:09:05.562740 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.562671    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 09:09:05.667146 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667116    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-socket\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667336 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667160    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-credential-socket\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667336 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667180    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-data\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667336 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667227    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-podinfo\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667336 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667243    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-envoy\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667336 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667315    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a925225-2eca-43a5-b4da-67857dcc22a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a925225-2eca-43a5-b4da-67857dcc22a3-istiod-ca-cert\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667590 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667396    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-kube-api-access-lgb7g\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-kube-api-access-lgb7g\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667590 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667455    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-token\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-token\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667590 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667424    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-socket" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:05.667590 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667495    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-credential-socket" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:05.667590 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667521    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-certs\") pod \"4a925225-2eca-43a5-b4da-67857dcc22a3\" (UID: \"4a925225-2eca-43a5-b4da-67857dcc22a3\") "
Nov 01 09:09:05.667842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667587    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-data" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:05.667842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667775    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a925225-2eca-43a5-b4da-67857dcc22a3-istiod-ca-cert" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:09:05.667949 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.667856    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-certs" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:05.668092 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.668077    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:05.668166 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.668098    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:05.668166 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.668116    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:05.668166 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.668130    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:05.668166 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.668144    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a925225-2eca-43a5-b4da-67857dcc22a3-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:05.670311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.670277    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-envoy" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:05.670311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.670305    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-podinfo" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:09:05.670455 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.670314    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-token" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:09:05.670455 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.670325    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-kube-api-access-lgb7g" pod "4a925225-2eca-43a5-b4da-67857dcc22a3" (UID: "4a925225-2eca-43a5-b4da-67857dcc22a3"). InnerVolumeSpecName "kube-api-access-lgb7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:09:05.775054 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.774982    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgb7g\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-kube-api-access-lgb7g\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:05.775054 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.775048    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:05.775377 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.775067    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:05.775377 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:05.775082    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a925225-2eca-43a5-b4da-67857dcc22a3-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:06.175984 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.175939    4931 generic.go:356] "Generic (PLEG): container finished" podID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerID="c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e" exitCode=0
Nov 01 09:09:06.175984 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.175987    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"
Nov 01 09:09:06.176251 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.176014    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" event={"ID":"4a925225-2eca-43a5-b4da-67857dcc22a3","Type":"ContainerDied","Data":"c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e"}
Nov 01 09:09:06.176251 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.176093    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln" event={"ID":"4a925225-2eca-43a5-b4da-67857dcc22a3","Type":"ContainerDied","Data":"306482324e663a55c31ae22e4fd6e2ee18f4b834e905d42cc6c2e34e55422379"}
Nov 01 09:09:06.176251 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.176114    4931 scope.go:122] "RemoveContainer" containerID="c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e"
Nov 01 09:09:06.186557 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.186534    4931 scope.go:122] "RemoveContainer" containerID="c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e"
Nov 01 09:09:06.186856 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:09:06.186837    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e\": container with ID starting with c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e not found: ID does not exist" containerID="c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e"
Nov 01 09:09:06.186915 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.186865    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e"} err="failed to get container status \"c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e\": rpc error: code = NotFound desc = could not find container \"c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e\": container with ID starting with c9aea12282ff1fdab0daa5f9fef3f93053990d1e0dd52dd27e656f9018a5ae2e not found: ID does not exist"
Nov 01 09:09:06.339748 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.339630    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"]
Nov 01 09:09:06.387267 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:06.387234    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-7g5ln"]
Nov 01 09:09:08.048442 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:08.048411    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" path="/var/lib/kubelet/pods/4a925225-2eca-43a5-b4da-67857dcc22a3/volumes"
Nov 01 09:09:15.393119 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:15.393072    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"]
Nov 01 09:09:15.393538 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:15.393391    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" containerID="cri-o://aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded" gracePeriod=30
Nov 01 09:09:15.446298 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:15.446267    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"]
Nov 01 09:09:15.446533 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:15.446497    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy" containerID="cri-o://8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8" gracePeriod=30
Nov 01 09:09:15.830190 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:15.830093    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-fvjhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:09:15.830190 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:15.830160    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:09:15.902394 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:15.902341    4931 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:09:15.902564 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:15.902411    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:09:20.739293 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.739263    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 09:09:20.793573 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.793539    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 09:09:20.884571 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884537    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-certs\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.884781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884609    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-certs\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.884781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884627    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-socket\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.884781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884655    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-podinfo\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.884781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884681    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-data\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.884781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884711    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-credential-socket\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.884781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884775    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-kube-api-access-qkq4h\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-kube-api-access-qkq4h\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884830    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-envoy\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884833    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-certs" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884854    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istiod-ca-cert\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884883    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istiod-ca-cert\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884937    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-kube-api-access-64spg\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-kube-api-access-64spg\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884984    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-token\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-token\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885004    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-token\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-token\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885054    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-data\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885075    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-envoy\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885100    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-socket\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885130    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-podinfo\") pod \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\" (UID: \"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2\") "
Nov 01 09:09:20.885153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885150    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-credential-socket\") pod \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\" (UID: \"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce\") "
Nov 01 09:09:20.885729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885625    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.885935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884880    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-socket" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.886013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884877    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-certs" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.886013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.884998    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-credential-socket" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.886013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885068    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-data" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.886013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885248    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istiod-ca-cert" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:09:20.886013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885680    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istiod-ca-cert" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:09:20.886013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.885843    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-credential-socket" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.886375 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.886250    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-socket" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.886375 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.886259    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-data" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.888317 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.888291    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-podinfo" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:09:20.888428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.888324    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-token" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:09:20.888428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.888376    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-kube-api-access-64spg" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "kube-api-access-64spg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:09:20.888709 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.888677    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-envoy" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.888964 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.888938    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-podinfo" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:09:20.889531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.889500    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-token" pod "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" (UID: "5f4d6ff6-1afe-430a-9977-f8e5e9f223a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:09:20.890282 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.890249    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-kube-api-access-qkq4h" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "kube-api-access-qkq4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:09:20.890367 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.890307    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-envoy" pod "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" (UID: "1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:20.986602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986518    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986548    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986558    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986571    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986583    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986592    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkq4h\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-kube-api-access-qkq4h\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986600    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986609    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986617    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986625    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64spg\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-kube-api-access-64spg\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986635    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986646    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986659    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986666    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986674    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986682    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:20.986927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:20.986690    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:21.320076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.319972    4931 generic.go:356] "Generic (PLEG): container finished" podID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerID="aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded" exitCode=0
Nov 01 09:09:21.320076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.320019    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"
Nov 01 09:09:21.320076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.320056    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" event={"ID":"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce","Type":"ContainerDied","Data":"aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded"}
Nov 01 09:09:21.320345 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.320116    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx" event={"ID":"1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce","Type":"ContainerDied","Data":"461d5afc6b163000dc2ba66a8a34d6db21b438bda0198c2cc2cb90caf6490e60"}
Nov 01 09:09:21.320345 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.320146    4931 scope.go:122] "RemoveContainer" containerID="aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded"
Nov 01 09:09:21.321770 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.321748    4931 generic.go:356] "Generic (PLEG): container finished" podID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerID="8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8" exitCode=0
Nov 01 09:09:21.321881 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.321793    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"
Nov 01 09:09:21.321881 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.321827    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" event={"ID":"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2","Type":"ContainerDied","Data":"8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8"}
Nov 01 09:09:21.321984 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.321890    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9" event={"ID":"5f4d6ff6-1afe-430a-9977-f8e5e9f223a2","Type":"ContainerDied","Data":"ac92795422f5ceed3e23f3d5e0dd22d78f6f5c017a2ff3f2abb9511c1a266914"}
Nov 01 09:09:21.332307 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.332286    4931 scope.go:122] "RemoveContainer" containerID="aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded"
Nov 01 09:09:21.332660 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:09:21.332633    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded\": container with ID starting with aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded not found: ID does not exist" containerID="aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded"
Nov 01 09:09:21.332763 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.332667    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded"} err="failed to get container status \"aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded\": rpc error: code = NotFound desc = could not find container \"aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded\": container with ID starting with aa5d63f311945b4f04f4f8ee8bb230a031b42b67b82aaf78785ac773e37a3ded not found: ID does not exist"
Nov 01 09:09:21.332763 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.332688    4931 scope.go:122] "RemoveContainer" containerID="8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8"
Nov 01 09:09:21.342270 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.342246    4931 scope.go:122] "RemoveContainer" containerID="8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8"
Nov 01 09:09:21.342533 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:09:21.342514    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8\": container with ID starting with 8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8 not found: ID does not exist" containerID="8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8"
Nov 01 09:09:21.342598 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.342539    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8"} err="failed to get container status \"8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8\": rpc error: code = NotFound desc = could not find container \"8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8\": container with ID starting with 8429927d732e15c6d40ab079dc75a916a8ef8a20e90ad7e79cfecf8f65e55ec8 not found: ID does not exist"
Nov 01 09:09:21.483229 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.483194    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"]
Nov 01 09:09:21.530482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.530448    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-f6qx9"]
Nov 01 09:09:21.636151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.636119    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"]
Nov 01 09:09:21.642043 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:21.642008    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-fvjhx"]
Nov 01 09:09:22.048391 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:22.048315    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" path="/var/lib/kubelet/pods/1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce/volumes"
Nov 01 09:09:22.048811 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:22.048796    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" path="/var/lib/kubelet/pods/5f4d6ff6-1afe-430a-9977-f8e5e9f223a2/volumes"
Nov 01 09:09:31.023608 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:09:31.023567    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:09:48.897136 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:48.897097    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"]
Nov 01 09:09:48.897666 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:48.897374    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" containerID="cri-o://303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156" gracePeriod=30
Nov 01 09:09:52.097227 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:52.097189    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:09:52.097705 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:52.097249    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:09:54.163309 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.163284    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 09:09:54.295164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295078    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-istio-token\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-istio-token\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295127    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-socket\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295157    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cecf2618-f684-49a2-adaf-6c5f84550877-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecf2618-f684-49a2-adaf-6c5f84550877-istiod-ca-cert\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295228    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-certs\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295296    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-kube-api-access-fkkgp\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295370    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-data\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295422    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-socket" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:54.295589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295488    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-credential-socket\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295528    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cecf2618-f684-49a2-adaf-6c5f84550877-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecf2618-f684-49a2-adaf-6c5f84550877-istio-podinfo\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295566    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-certs" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:54.295800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295595    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-envoy\") pod \"cecf2618-f684-49a2-adaf-6c5f84550877\" (UID: \"cecf2618-f684-49a2-adaf-6c5f84550877\") "
Nov 01 09:09:54.295800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295732    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecf2618-f684-49a2-adaf-6c5f84550877-istiod-ca-cert" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:09:54.295800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295731    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-data" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:54.295800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.295762    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-credential-socket" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:54.296164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.296144    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.296452 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.296169    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecf2618-f684-49a2-adaf-6c5f84550877-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.296452 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.296179    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.296452 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.296188    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.296452 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.296197    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.297677 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.297648    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-istio-token" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:09:54.298060 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.298009    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-envoy" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:09:54.298264 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.298245    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cecf2618-f684-49a2-adaf-6c5f84550877-istio-podinfo" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:09:54.298318 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.298266    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-kube-api-access-fkkgp" pod "cecf2618-f684-49a2-adaf-6c5f84550877" (UID: "cecf2618-f684-49a2-adaf-6c5f84550877"). InnerVolumeSpecName "kube-api-access-fkkgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:09:54.397865 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.397827    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecf2618-f684-49a2-adaf-6c5f84550877-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.397865 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.397857    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecf2618-f684-49a2-adaf-6c5f84550877-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.397865 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.397867    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.398149 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.397876    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/cecf2618-f684-49a2-adaf-6c5f84550877-kube-api-access-fkkgp\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:09:54.553288 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.553202    4931 generic.go:356] "Generic (PLEG): container finished" podID="cecf2618-f684-49a2-adaf-6c5f84550877" containerID="303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156" exitCode=0
Nov 01 09:09:54.553288 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.553255    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"
Nov 01 09:09:54.553288 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.553275    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" event={"ID":"cecf2618-f684-49a2-adaf-6c5f84550877","Type":"ContainerDied","Data":"303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156"}
Nov 01 09:09:54.553558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.553333    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws" event={"ID":"cecf2618-f684-49a2-adaf-6c5f84550877","Type":"ContainerDied","Data":"8a6b9b07bd1e81e68b53d7293eb532124053b816472b47753b2e839c95bc6ebc"}
Nov 01 09:09:54.553558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.553348    4931 scope.go:122] "RemoveContainer" containerID="303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156"
Nov 01 09:09:54.564785 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.564759    4931 scope.go:122] "RemoveContainer" containerID="303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156"
Nov 01 09:09:54.565167 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:09:54.565144    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156\": container with ID starting with 303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156 not found: ID does not exist" containerID="303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156"
Nov 01 09:09:54.565254 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.565179    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156"} err="failed to get container status \"303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156\": rpc error: code = NotFound desc = could not find container \"303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156\": container with ID starting with 303f92a28bf97b34d7496dff78b4c26385d55db858d8aef99780de6a55936156 not found: ID does not exist"
Nov 01 09:09:54.712912 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.712864    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"]
Nov 01 09:09:54.760802 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:54.760766    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-njcws"]
Nov 01 09:09:56.047900 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:09:56.047865    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" path="/var/lib/kubelet/pods/cecf2618-f684-49a2-adaf-6c5f84550877/volumes"
Nov 01 09:10:01.023316 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:10:01.023277    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:10:10.146540 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.146506    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"]
Nov 01 09:10:10.147307 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147287    4931 cpu_manager.go:414] "containerMap: removing container" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy"
Nov 01 09:10:10.147370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147311    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy"
Nov 01 09:10:10.147370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147332    4931 cpu_manager.go:414] "containerMap: removing container" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy"
Nov 01 09:10:10.147370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147340    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy"
Nov 01 09:10:10.147468 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147371    4931 cpu_manager.go:414] "containerMap: removing container" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy"
Nov 01 09:10:10.147468 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147379    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy"
Nov 01 09:10:10.147468 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147409    4931 cpu_manager.go:414] "containerMap: removing container" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy"
Nov 01 09:10:10.147468 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147417    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy"
Nov 01 09:10:10.147602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147568    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecf2618-f684-49a2-adaf-6c5f84550877" containerName="istio-proxy"
Nov 01 09:10:10.147602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147585    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7ae9a2-9efa-4c86-91f3-3c84f1d1bdce" containerName="istio-proxy"
Nov 01 09:10:10.147602 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147596    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4d6ff6-1afe-430a-9977-f8e5e9f223a2" containerName="istio-proxy"
Nov 01 09:10:10.147707 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.147603    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a925225-2eca-43a5-b4da-67857dcc22a3" containerName="istio-proxy"
Nov 01 09:10:10.151571 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.151551    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.204558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.204530    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"]
Nov 01 09:10:10.248303 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.248269    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c3aaa7-serving-cert\""
Nov 01 09:10:10.285664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.285627    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.285869 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.285794    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.285985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.285959    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-metrics-certs\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.286154 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.286134    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qst\" (UniqueName: \"kubernetes.io/projected/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-kube-api-access-g8qst\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.387576 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.387545    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qst\" (UniqueName: \"kubernetes.io/projected/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-kube-api-access-g8qst\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.387769 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.387592    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.387769 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.387646    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.387769 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.387714    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-metrics-certs\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.387938 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:10:10.387774    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-serving-cert: secret "maas-api-e2e-praxis-a-c3aaa7-serving-cert" not found
Nov 01 09:10:10.387938 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:10:10.387854    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls podName:be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a nodeName:}" failed. No retries permitted until 2026-11-01 09:10:10.887835923 +0000 UTC m=+2281.002708889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls") pod "maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg" (UID: "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a") : secret "maas-api-e2e-praxis-a-c3aaa7-serving-cert" not found
Nov 01 09:10:10.388470 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.388446    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.390413 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.390395    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-metrics-certs\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.449627 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.449556    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qst\" (UniqueName: \"kubernetes.io/projected/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-kube-api-access-g8qst\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.893564 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.893529    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:10.896359 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:10.896330    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:11.067814 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:11.067776    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:11.459759 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:11.459728    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"]
Nov 01 09:10:11.460536 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:10:11.460506    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe37e0e4_ef60_4fe8_8ecb_7ab8b1239b2a.slice/crio-d8363a60f78ce5221616e40655981945eeaa11858742984639ea39198ae52be3 WatchSource:0}: Error finding container d8363a60f78ce5221616e40655981945eeaa11858742984639ea39198ae52be3: Status 404 returned error can't find the container with id d8363a60f78ce5221616e40655981945eeaa11858742984639ea39198ae52be3
Nov 01 09:10:11.673350 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:11.673316    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg" event={"ID":"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a","Type":"ContainerStarted","Data":"d8363a60f78ce5221616e40655981945eeaa11858742984639ea39198ae52be3"}
Nov 01 09:10:12.684607 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:12.684564    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg" event={"ID":"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a","Type":"ContainerStarted","Data":"694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d"}
Nov 01 09:10:12.685146 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:12.684701    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:12.798947 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:12.798840    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg" podStartSLOduration=1.818224921 podStartE2EDuration="2.798824387s" totalImagesPullingTime="980.599466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:10:12.798134895 +0000 UTC m=+2282.913007864" watchObservedRunningTime="2026-11-01 09:10:12.798824387 +0000 UTC m=+2282.913697356"
Nov 01 09:10:18.698750 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:10:18.698718    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:10:31.023420 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:10:31.023340    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:11:01.024739 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:11:01.024690    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:11:31.024044 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:11:31.023992    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:11:59.811690 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:11:59.811603    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"]
Nov 01 09:11:59.812235 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:11:59.811859    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw" podUID="466367d6-f226-4d59-bbe1-8b5ffc058318" containerName="maas-api" containerID="cri-o://21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8" gracePeriod=30
Nov 01 09:12:00.154479 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.154441    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:12:00.281600 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.281563    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-maas-api-tls\") pod \"466367d6-f226-4d59-bbe1-8b5ffc058318\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") "
Nov 01 09:12:00.281882 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.281867    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-metrics-certs\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-metrics-certs\") pod \"466367d6-f226-4d59-bbe1-8b5ffc058318\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") "
Nov 01 09:12:00.282096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.282064    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466367d6-f226-4d59-bbe1-8b5ffc058318-kube-api-access-7rppj\" (UniqueName: \"kubernetes.io/projected/466367d6-f226-4d59-bbe1-8b5ffc058318-kube-api-access-7rppj\") pod \"466367d6-f226-4d59-bbe1-8b5ffc058318\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") "
Nov 01 09:12:00.282306 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.282291    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466367d6-f226-4d59-bbe1-8b5ffc058318-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466367d6-f226-4d59-bbe1-8b5ffc058318-openshift-service-ca\") pod \"466367d6-f226-4d59-bbe1-8b5ffc058318\" (UID: \"466367d6-f226-4d59-bbe1-8b5ffc058318\") "
Nov 01 09:12:00.283290 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.283264    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466367d6-f226-4d59-bbe1-8b5ffc058318-openshift-service-ca" pod "466367d6-f226-4d59-bbe1-8b5ffc058318" (UID: "466367d6-f226-4d59-bbe1-8b5ffc058318"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:12:00.311626 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.311289    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466367d6-f226-4d59-bbe1-8b5ffc058318-kube-api-access-7rppj" pod "466367d6-f226-4d59-bbe1-8b5ffc058318" (UID: "466367d6-f226-4d59-bbe1-8b5ffc058318"). InnerVolumeSpecName "kube-api-access-7rppj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:12:00.311626 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.311437    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-maas-api-tls" pod "466367d6-f226-4d59-bbe1-8b5ffc058318" (UID: "466367d6-f226-4d59-bbe1-8b5ffc058318"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:12:00.311626 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.311559    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-metrics-certs" pod "466367d6-f226-4d59-bbe1-8b5ffc058318" (UID: "466367d6-f226-4d59-bbe1-8b5ffc058318"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:12:00.384415 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.384332    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466367d6-f226-4d59-bbe1-8b5ffc058318-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:00.384415 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.384366    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:00.384415 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.384375    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466367d6-f226-4d59-bbe1-8b5ffc058318-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:00.384415 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.384384    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rppj\" (UniqueName: \"kubernetes.io/projected/466367d6-f226-4d59-bbe1-8b5ffc058318-kube-api-access-7rppj\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:12:00.440157 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.440123    4931 generic.go:356] "Generic (PLEG): container finished" podID="466367d6-f226-4d59-bbe1-8b5ffc058318" containerID="21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8" exitCode=0
Nov 01 09:12:00.440341 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.440173    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"
Nov 01 09:12:00.440341 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.440213    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw" event={"ID":"466367d6-f226-4d59-bbe1-8b5ffc058318","Type":"ContainerDied","Data":"21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8"}
Nov 01 09:12:00.440341 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.440266    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw" event={"ID":"466367d6-f226-4d59-bbe1-8b5ffc058318","Type":"ContainerDied","Data":"5e1bf3a59033a4048b870c3427fbf106b34ea3668b397ac30f629a7668deb75c"}
Nov 01 09:12:00.440341 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.440289    4931 scope.go:122] "RemoveContainer" containerID="21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8"
Nov 01 09:12:00.450892 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.450874    4931 scope.go:122] "RemoveContainer" containerID="21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8"
Nov 01 09:12:00.451232 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:12:00.451209    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8\": container with ID starting with 21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8 not found: ID does not exist" containerID="21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8"
Nov 01 09:12:00.451338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.451240    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8"} err="failed to get container status \"21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8\": rpc error: code = NotFound desc = could not find container \"21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8\": container with ID starting with 21292e4f01652e89c83af9784a7bad5505eef9d16c9fd0232b49066741658be8 not found: ID does not exist"
Nov 01 09:12:00.599324 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.599291    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"]
Nov 01 09:12:00.603525 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:00.603497    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0460d-cf7f787d-854vw"]
Nov 01 09:12:01.022751 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:12:01.022714    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:12:02.056678 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:02.056638    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466367d6-f226-4d59-bbe1-8b5ffc058318" path="/var/lib/kubelet/pods/466367d6-f226-4d59-bbe1-8b5ffc058318/volumes"
Nov 01 09:12:10.483532 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:10.483488    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:12:10.488117 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:10.488090    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:12:10.512774 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:10.512743    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:12:10.520930 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:10.520893    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:12:31.022627 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:12:31.022588    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:12:35.714838 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.714800    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"]
Nov 01 09:12:35.715593 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.715573    4931 cpu_manager.go:414] "containerMap: removing container" podUID="466367d6-f226-4d59-bbe1-8b5ffc058318" containerName="maas-api"
Nov 01 09:12:35.715637 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.715597    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466367d6-f226-4d59-bbe1-8b5ffc058318" containerName="maas-api"
Nov 01 09:12:35.715732 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.715721    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="466367d6-f226-4d59-bbe1-8b5ffc058318" containerName="maas-api"
Nov 01 09:12:35.719066 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.719050    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.816242 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.816197    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.816466 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.816295    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.816466 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.816394    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-metrics-certs\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.816610 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.816511    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb94m\" (UniqueName: \"kubernetes.io/projected/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-kube-api-access-mb94m\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.819266 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.819239    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-55644e-serving-cert\""
Nov 01 09:12:35.822378 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.822353    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"]
Nov 01 09:12:35.917996 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.917955    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.918204 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.918005    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.918204 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.918127    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-metrics-certs\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.918204 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:12:35.918138    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-serving-cert: secret "maas-api-e2e-praxis-b-55644e-serving-cert" not found
Nov 01 09:12:35.918204 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.918159    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb94m\" (UniqueName: \"kubernetes.io/projected/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-kube-api-access-mb94m\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.918433 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:12:35.918211    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls podName:0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8 nodeName:}" failed. No retries permitted until 2026-11-01 09:12:36.418195436 +0000 UTC m=+2426.533068382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls") pod "maas-api-e2e-praxis-b-55644e-7f658d674-gpnld" (UID: "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8") : secret "maas-api-e2e-praxis-b-55644e-serving-cert" not found
Nov 01 09:12:35.918704 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.918683    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.920733 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.920712    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-metrics-certs\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:35.994612 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:35.994521    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb94m\" (UniqueName: \"kubernetes.io/projected/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-kube-api-access-mb94m\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:36.424654 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:36.424614    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:36.427402 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:36.427379    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-55644e-7f658d674-gpnld\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:36.632942 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:36.632893    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:37.008206 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:37.008173    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"]
Nov 01 09:12:37.008583 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:12:37.008308    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0388dcc9_9cc9_42f8_af9d_5762f9ac4dc8.slice/crio-6c011c84f0545a57d435fbadb6cca6e9765b8b20a2809de78535c4f5812784b2 WatchSource:0}: Error finding container 6c011c84f0545a57d435fbadb6cca6e9765b8b20a2809de78535c4f5812784b2: Status 404 returned error can't find the container with id 6c011c84f0545a57d435fbadb6cca6e9765b8b20a2809de78535c4f5812784b2
Nov 01 09:12:37.010255 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:37.010233    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:12:37.729921 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:37.729874    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld" event={"ID":"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8","Type":"ContainerStarted","Data":"6c011c84f0545a57d435fbadb6cca6e9765b8b20a2809de78535c4f5812784b2"}
Nov 01 09:12:38.737496 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:38.737422    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld" event={"ID":"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8","Type":"ContainerStarted","Data":"5419aaaaf31da14d96bfa00ba52a0cd92979a3ff2961d5c6bbb37d0fdde2b08d"}
Nov 01 09:12:38.737965 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:38.737659    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:12:38.851929 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:38.851873    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld" podStartSLOduration=2.962329296 podStartE2EDuration="3.851859473s" totalImagesPullingTime="889.530177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:12:38.85049557 +0000 UTC m=+2428.965368541" watchObservedRunningTime="2026-11-01 09:12:38.851859473 +0000 UTC m=+2428.966732440"
Nov 01 09:12:44.749814 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:12:44.749785    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:13:01.036049 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:13:01.035985    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:13:07.023899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.023862    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"]
Nov 01 09:13:07.027991 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.027967    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.077736 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.077702    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xc4\" (UniqueName: \"kubernetes.io/projected/7241d569-f842-40c9-b66f-78a784161a71-kube-api-access-27xc4\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.077931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.077753    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7241d569-f842-40c9-b66f-78a784161a71-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.077931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.077833    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.077931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.077915    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-metrics-certs\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.079501 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.079476    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-28efa1-serving-cert\""
Nov 01 09:13:07.085932 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.085908    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"]
Nov 01 09:13:07.179137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.179098    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xc4\" (UniqueName: \"kubernetes.io/projected/7241d569-f842-40c9-b66f-78a784161a71-kube-api-access-27xc4\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.179333 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.179153    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7241d569-f842-40c9-b66f-78a784161a71-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.179333 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.179209    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.179464 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:13:07.179376    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-serving-cert: secret "maas-api-e2e-praxis-a-28efa1-serving-cert" not found
Nov 01 09:13:07.179464 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:13:07.179448    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls podName:7241d569-f842-40c9-b66f-78a784161a71 nodeName:}" failed. No retries permitted until 2026-11-01 09:13:07.679427649 +0000 UTC m=+2457.794300610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls") pod "maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq" (UID: "7241d569-f842-40c9-b66f-78a784161a71") : secret "maas-api-e2e-praxis-a-28efa1-serving-cert" not found
Nov 01 09:13:07.179583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.179539    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-metrics-certs\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.179928 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.179901    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7241d569-f842-40c9-b66f-78a784161a71-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.182310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.182292    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-metrics-certs\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.240339 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.240309    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xc4\" (UniqueName: \"kubernetes.io/projected/7241d569-f842-40c9-b66f-78a784161a71-kube-api-access-27xc4\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.685318 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.685282    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.688011 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.687985    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls\") pod \"maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:07.941267 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:07.941185    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:08.316079 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:08.316051    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"]
Nov 01 09:13:08.317342 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:13:08.317314    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7241d569_f842_40c9_b66f_78a784161a71.slice/crio-66e633a053645953b55dac54069ad9b1a825261f852deec948b05c42cfdeffa8 WatchSource:0}: Error finding container 66e633a053645953b55dac54069ad9b1a825261f852deec948b05c42cfdeffa8: Status 404 returned error can't find the container with id 66e633a053645953b55dac54069ad9b1a825261f852deec948b05c42cfdeffa8
Nov 01 09:13:08.928974 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:08.928937    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq" event={"ID":"7241d569-f842-40c9-b66f-78a784161a71","Type":"ContainerStarted","Data":"66e633a053645953b55dac54069ad9b1a825261f852deec948b05c42cfdeffa8"}
Nov 01 09:13:09.939115 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:09.939076    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq" event={"ID":"7241d569-f842-40c9-b66f-78a784161a71","Type":"ContainerStarted","Data":"87941460a892022579080c086e396f447d4b799ab78be89509be650d6ffbe889"}
Nov 01 09:13:09.939585 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:09.939311    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:10.053170 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:10.053109    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq" podStartSLOduration=3.118095596 podStartE2EDuration="4.05309193s" totalImagesPullingTime="934.996334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:13:10.051364654 +0000 UTC m=+2460.166237622" watchObservedRunningTime="2026-11-01 09:13:10.05309193 +0000 UTC m=+2460.167964900"
Nov 01 09:13:15.953134 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:13:15.953098    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:13:31.024757 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:13:31.024670    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:14:01.023707 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:14:01.023661    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:14:31.027156 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:14:31.027116    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:14:36.063045 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:36.062999    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"]
Nov 01 09:14:36.063512 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:36.063268    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy" containerID="cri-o://e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef" gracePeriod=30
Nov 01 09:14:37.624821 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.624782    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"]
Nov 01 09:14:37.625414 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.625106    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg" podUID="be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" containerName="maas-api" containerID="cri-o://694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d" gracePeriod=30
Nov 01 09:14:37.932690 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.932658    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:14:37.976935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.976895    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-openshift-service-ca\") pod \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") "
Nov 01 09:14:37.977146 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.976991    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-metrics-certs\") pod \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") "
Nov 01 09:14:37.977146 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.977093    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls\") pod \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") "
Nov 01 09:14:37.977279 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.977231    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-kube-api-access-g8qst\" (UniqueName: \"kubernetes.io/projected/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-kube-api-access-g8qst\") pod \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\" (UID: \"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a\") "
Nov 01 09:14:37.977345 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.977294    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-openshift-service-ca" pod "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" (UID: "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:14:37.977644 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.977621    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:37.980384 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.980353    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-metrics-certs" pod "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" (UID: "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:14:37.980503 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.980421    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls" pod "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" (UID: "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:14:37.981471 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:37.981447    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-kube-api-access-g8qst" pod "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" (UID: "be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a"). InnerVolumeSpecName "kube-api-access-g8qst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:38.079330 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.079078    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qst\" (UniqueName: \"kubernetes.io/projected/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-kube-api-access-g8qst\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:38.079330 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.079195    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:38.079330 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.079213    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:38.622770 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.622725    4931 generic.go:356] "Generic (PLEG): container finished" podID="be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" containerID="694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d" exitCode=0
Nov 01 09:14:38.622967 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.622785    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"
Nov 01 09:14:38.622967 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.622810    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg" event={"ID":"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a","Type":"ContainerDied","Data":"694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d"}
Nov 01 09:14:38.622967 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.622878    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg" event={"ID":"be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a","Type":"ContainerDied","Data":"d8363a60f78ce5221616e40655981945eeaa11858742984639ea39198ae52be3"}
Nov 01 09:14:38.622967 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.622908    4931 scope.go:122] "RemoveContainer" containerID="694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d"
Nov 01 09:14:38.634727 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.634550    4931 scope.go:122] "RemoveContainer" containerID="694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d"
Nov 01 09:14:38.634982 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:14:38.634829    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d\": container with ID starting with 694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d not found: ID does not exist" containerID="694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d"
Nov 01 09:14:38.634982 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.634855    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d"} err="failed to get container status \"694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d\": rpc error: code = NotFound desc = could not find container \"694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d\": container with ID starting with 694952c4ce1d413fffb22a77980e014e18daed47c8f121ff34022ddd9fcbba5d not found: ID does not exist"
Nov 01 09:14:38.788811 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.788777    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"]
Nov 01 09:14:38.791601 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:38.791579    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c3aaa7-786d944786-bzmxg"]
Nov 01 09:14:40.049384 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:40.049355    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" path="/var/lib/kubelet/pods/be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a/volumes"
Nov 01 09:14:41.332446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.332376    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 09:14:41.430622 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430590    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-socket\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.430809 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430688    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-credential-socket\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.430809 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430739    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-kube-api-access-2sfzk\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-kube-api-access-2sfzk\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.430809 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430787    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-envoy\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.430968 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430832    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-token\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-token\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.430968 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430874    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-podinfo\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.430968 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430902    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-socket" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.430968 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430940    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-certs\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.431186 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.430984    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-credential-socket" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.431186 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431090    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-data\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.431186 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431139    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istiod-ca-cert\") pod \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\" (UID: \"d450b25e-ae73-4dac-9f39-9ee97becdd4a\") "
Nov 01 09:14:41.431344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431266    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-certs" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.431344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431314    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-data" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.431689 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431617    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.431689 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431635    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istiod-ca-cert" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:14:41.431689 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431641    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.432110 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431697    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.432110 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.431732    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.433518 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.433479    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-podinfo" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:14:41.433688 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.433665    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-kube-api-access-2sfzk" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "kube-api-access-2sfzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:41.433806 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.433740    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-envoy" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:14:41.433876 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.433797    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-token" pod "d450b25e-ae73-4dac-9f39-9ee97becdd4a" (UID: "d450b25e-ae73-4dac-9f39-9ee97becdd4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:14:41.533073 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.533040    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.533073 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.533071    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.533073 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.533080    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.533295 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.533089    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d450b25e-ae73-4dac-9f39-9ee97becdd4a-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.533295 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.533097    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sfzk\" (UniqueName: \"kubernetes.io/projected/d450b25e-ae73-4dac-9f39-9ee97becdd4a-kube-api-access-2sfzk\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:14:41.652417 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.652380    4931 generic.go:356] "Generic (PLEG): container finished" podID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerID="e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef" exitCode=0
Nov 01 09:14:41.652570 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.652433    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"
Nov 01 09:14:41.652570 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.652468    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" event={"ID":"d450b25e-ae73-4dac-9f39-9ee97becdd4a","Type":"ContainerDied","Data":"e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef"}
Nov 01 09:14:41.652570 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.652552    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql" event={"ID":"d450b25e-ae73-4dac-9f39-9ee97becdd4a","Type":"ContainerDied","Data":"5c423d4aebe1e724d925d96f989a836bda2f012db0343ff0cc7429a24b5c9a1d"}
Nov 01 09:14:41.652708 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.652575    4931 scope.go:122] "RemoveContainer" containerID="e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef"
Nov 01 09:14:41.663369 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.663353    4931 scope.go:122] "RemoveContainer" containerID="e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef"
Nov 01 09:14:41.663629 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:14:41.663612    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef\": container with ID starting with e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef not found: ID does not exist" containerID="e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef"
Nov 01 09:14:41.663668 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.663641    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef"} err="failed to get container status \"e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef\": rpc error: code = NotFound desc = could not find container \"e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef\": container with ID starting with e1f943a0f972b97c3382472fd8935f3722d5f09c520d93f8e545d4aa806afdef not found: ID does not exist"
Nov 01 09:14:41.812384 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.812345    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"]
Nov 01 09:14:41.817363 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:41.817337    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-drvql"]
Nov 01 09:14:42.048979 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:14:42.048889    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" path="/var/lib/kubelet/pods/d450b25e-ae73-4dac-9f39-9ee97becdd4a/volumes"
Nov 01 09:15:01.026325 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:15:01.026239    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:15:31.023549 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:15:31.023510    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:15:42.554179 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.554137    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"]
Nov 01 09:15:42.554786 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.554376    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w" podUID="e1cdc679-f48f-4f89-bec2-1048ab60fbcb" containerName="maas-api" containerID="cri-o://cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6" gracePeriod=30
Nov 01 09:15:42.643096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.643017    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w" podUID="e1cdc679-f48f-4f89-bec2-1048ab60fbcb" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.98:8443/health\": dial tcp 10.133.0.98:8443: connect: connection refused"
Nov 01 09:15:42.898062 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.898017    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:15:42.978617 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.978582    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-metrics-certs\") pod \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") "
Nov 01 09:15:42.978815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.978667    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls\") pod \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") "
Nov 01 09:15:42.978815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.978717    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-openshift-service-ca\") pod \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") "
Nov 01 09:15:42.978948 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.978925    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-kube-api-access-c4j6t\" (UniqueName: \"kubernetes.io/projected/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-kube-api-access-c4j6t\") pod \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\" (UID: \"e1cdc679-f48f-4f89-bec2-1048ab60fbcb\") "
Nov 01 09:15:42.979192 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.979147    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-openshift-service-ca" pod "e1cdc679-f48f-4f89-bec2-1048ab60fbcb" (UID: "e1cdc679-f48f-4f89-bec2-1048ab60fbcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:42.979672 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.979651    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:42.981575 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.981532    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls" pod "e1cdc679-f48f-4f89-bec2-1048ab60fbcb" (UID: "e1cdc679-f48f-4f89-bec2-1048ab60fbcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:15:42.981682 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.981618    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-metrics-certs" pod "e1cdc679-f48f-4f89-bec2-1048ab60fbcb" (UID: "e1cdc679-f48f-4f89-bec2-1048ab60fbcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:15:42.981861 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:42.981833    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-kube-api-access-c4j6t" pod "e1cdc679-f48f-4f89-bec2-1048ab60fbcb" (UID: "e1cdc679-f48f-4f89-bec2-1048ab60fbcb"). InnerVolumeSpecName "kube-api-access-c4j6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:43.080869 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.080838    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:43.080869 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.080874    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4j6t\" (UniqueName: \"kubernetes.io/projected/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-kube-api-access-c4j6t\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:43.081167 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.080887    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cdc679-f48f-4f89-bec2-1048ab60fbcb-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:43.087612 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.087579    4931 generic.go:356] "Generic (PLEG): container finished" podID="e1cdc679-f48f-4f89-bec2-1048ab60fbcb" containerID="cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6" exitCode=0
Nov 01 09:15:43.087755 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.087627    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"
Nov 01 09:15:43.087755 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.087661    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w" event={"ID":"e1cdc679-f48f-4f89-bec2-1048ab60fbcb","Type":"ContainerDied","Data":"cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6"}
Nov 01 09:15:43.087755 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.087721    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w" event={"ID":"e1cdc679-f48f-4f89-bec2-1048ab60fbcb","Type":"ContainerDied","Data":"340ba2ea73ac304304024caba08069123e6e046530f508a80787b30108ef4d6b"}
Nov 01 09:15:43.087755 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.087749    4931 scope.go:122] "RemoveContainer" containerID="cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6"
Nov 01 09:15:43.102868 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.102849    4931 scope.go:122] "RemoveContainer" containerID="cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6"
Nov 01 09:15:43.103160 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:15:43.103141    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6\": container with ID starting with cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6 not found: ID does not exist" containerID="cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6"
Nov 01 09:15:43.103223 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.103171    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6"} err="failed to get container status \"cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6\": rpc error: code = NotFound desc = could not find container \"cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6\": container with ID starting with cf870357ebae1fcaaea1e7a984f08a75b9f06fbd9523fe1721152da14f6475c6 not found: ID does not exist"
Nov 01 09:15:43.246377 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.246285    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"]
Nov 01 09:15:43.250507 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.250481    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be7cc-5d96b5fb75-v947w"]
Nov 01 09:15:43.753139 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.753099    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"]
Nov 01 09:15:43.753583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:43.753317    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld" podUID="0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" containerName="maas-api" containerID="cri-o://5419aaaaf31da14d96bfa00ba52a0cd92979a3ff2961d5c6bbb37d0fdde2b08d" gracePeriod=30
Nov 01 09:15:44.105483 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.105451    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1cdc679-f48f-4f89-bec2-1048ab60fbcb" path="/var/lib/kubelet/pods/e1cdc679-f48f-4f89-bec2-1048ab60fbcb/volumes"
Nov 01 09:15:44.109922 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.109891    4931 generic.go:356] "Generic (PLEG): container finished" podID="0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" containerID="5419aaaaf31da14d96bfa00ba52a0cd92979a3ff2961d5c6bbb37d0fdde2b08d" exitCode=0
Nov 01 09:15:44.110113 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.109966    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld" event={"ID":"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8","Type":"ContainerDied","Data":"5419aaaaf31da14d96bfa00ba52a0cd92979a3ff2961d5c6bbb37d0fdde2b08d"}
Nov 01 09:15:44.121393 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.121369    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:15:44.298422 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.298395    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-openshift-service-ca\") pod \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") "
Nov 01 09:15:44.298592 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.298528    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-kube-api-access-mb94m\" (UniqueName: \"kubernetes.io/projected/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-kube-api-access-mb94m\") pod \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") "
Nov 01 09:15:44.298592 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.298565    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls\") pod \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") "
Nov 01 09:15:44.298689 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.298638    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-metrics-certs\") pod \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\" (UID: \"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8\") "
Nov 01 09:15:44.298851 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.298812    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-openshift-service-ca" pod "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" (UID: "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:15:44.299118 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.299091    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:44.300992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.300960    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls" pod "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" (UID: "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:15:44.301126 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.301071    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-metrics-certs" pod "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" (UID: "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:15:44.301126 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.301096    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-kube-api-access-mb94m" pod "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" (UID: "0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8"). InnerVolumeSpecName "kube-api-access-mb94m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:15:44.400774 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.400738    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:44.400774 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.400768    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb94m\" (UniqueName: \"kubernetes.io/projected/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-kube-api-access-mb94m\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:44.400774 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:44.400778    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:15:45.117922 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:45.117887    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"
Nov 01 09:15:45.118459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:45.117927    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld" event={"ID":"0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8","Type":"ContainerDied","Data":"6c011c84f0545a57d435fbadb6cca6e9765b8b20a2809de78535c4f5812784b2"}
Nov 01 09:15:45.118459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:45.118005    4931 scope.go:122] "RemoveContainer" containerID="5419aaaaf31da14d96bfa00ba52a0cd92979a3ff2961d5c6bbb37d0fdde2b08d"
Nov 01 09:15:45.278214 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:45.278179    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"]
Nov 01 09:15:45.323887 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:45.323852    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-55644e-7f658d674-gpnld"]
Nov 01 09:15:46.047764 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:15:46.047726    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" path="/var/lib/kubelet/pods/0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8/volumes"
Nov 01 09:16:01.023578 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:16:01.023540    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:16:31.027293 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:16:31.027178    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:16:52.087344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087309    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"]
Nov 01 09:16:52.087815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087803    4931 cpu_manager.go:414] "containerMap: removing container" podUID="0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" containerName="maas-api"
Nov 01 09:16:52.087859 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087816    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" containerName="maas-api"
Nov 01 09:16:52.087859 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087830    4931 cpu_manager.go:414] "containerMap: removing container" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy"
Nov 01 09:16:52.087859 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087835    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy"
Nov 01 09:16:52.087957 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087867    4931 cpu_manager.go:414] "containerMap: removing container" podUID="be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" containerName="maas-api"
Nov 01 09:16:52.087957 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087872    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" containerName="maas-api"
Nov 01 09:16:52.087957 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087883    4931 cpu_manager.go:414] "containerMap: removing container" podUID="e1cdc679-f48f-4f89-bec2-1048ab60fbcb" containerName="maas-api"
Nov 01 09:16:52.087957 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087887    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1cdc679-f48f-4f89-bec2-1048ab60fbcb" containerName="maas-api"
Nov 01 09:16:52.088096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087965    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d450b25e-ae73-4dac-9f39-9ee97becdd4a" containerName="istio-proxy"
Nov 01 09:16:52.088096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087976    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="be37e0e4-ef60-4fe8-8ecb-7ab8b1239b2a" containerName="maas-api"
Nov 01 09:16:52.088096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087983    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1cdc679-f48f-4f89-bec2-1048ab60fbcb" containerName="maas-api"
Nov 01 09:16:52.088096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.087993    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0388dcc9-9cc9-42f8-af9d-5762f9ac4dc8" containerName="maas-api"
Nov 01 09:16:52.091491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.091469    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.193988 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.193947    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"]
Nov 01 09:16:52.210158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.210114    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b711ae-openshift-default-dockercfg-ms66n\""
Nov 01 09:16:52.226684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.226649    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.226866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.226691    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.226866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.226730    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.226866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.226821    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.226866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.226854    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6728c86f-880f-4ea6-8755-7bc9c01d4981-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.227041 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.226932    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.227041 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.226958    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmh8p\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-kube-api-access-qmh8p\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.227116 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.227018    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.227116 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.227073    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.331839 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.331784    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.332185 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.332163    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.332329 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.332315    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6728c86f-880f-4ea6-8755-7bc9c01d4981-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.332515 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.332496    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.332626 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.332597    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.332692 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.332366    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.332790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.332773    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmh8p\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-kube-api-access-qmh8p\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.332926 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.332896    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.333112 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.333092    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.333316 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.333298    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.333482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.333463    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.334697 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.334671    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.335555 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.335533    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6728c86f-880f-4ea6-8755-7bc9c01d4981-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.336399 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.336377    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.353114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.339091    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.353114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.341923    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.393748 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.393713    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmh8p\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-kube-api-access-qmh8p\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.394678 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.394654    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.407753 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.407712    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:52.839177 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:16:52.839130    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6728c86f_880f_4ea6_8755_7bc9c01d4981.slice/crio-53ef8d4a80e7b9af6a5910a113dd214b27bec602b419fe7da66f3146ddc74adb WatchSource:0}: Error finding container 53ef8d4a80e7b9af6a5910a113dd214b27bec602b419fe7da66f3146ddc74adb: Status 404 returned error can't find the container with id 53ef8d4a80e7b9af6a5910a113dd214b27bec602b419fe7da66f3146ddc74adb
Nov 01 09:16:52.839803 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.839775    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"]
Nov 01 09:16:52.842197 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.842160    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:16:52.842311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.842247    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:16:52.842311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:52.842288    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:16:53.636891 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:53.636845    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" event={"ID":"6728c86f-880f-4ea6-8755-7bc9c01d4981","Type":"ContainerStarted","Data":"2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d"}
Nov 01 09:16:53.636891 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:53.636896    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" event={"ID":"6728c86f-880f-4ea6-8755-7bc9c01d4981","Type":"ContainerStarted","Data":"53ef8d4a80e7b9af6a5910a113dd214b27bec602b419fe7da66f3146ddc74adb"}
Nov 01 09:16:53.861057 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:53.860343    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podStartSLOduration=1.860323004 podStartE2EDuration="1.860323004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:53.810449008 +0000 UTC m=+2683.925321977" watchObservedRunningTime="2026-11-01 09:16:53.860323004 +0000 UTC m=+2683.975195972"
Nov 01 09:16:54.408781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:54.408745    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:16:54.410092 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:54.410061    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 09:16:54.410238 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:54.410114    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 09:16:55.408698 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:55.408655    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 09:16:55.409144 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:55.408732    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 09:16:55.481050 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:55.470653    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"]
Nov 01 09:16:55.481050 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:55.470921    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq" podUID="7241d569-f842-40c9-b66f-78a784161a71" containerName="maas-api" containerID="cri-o://87941460a892022579080c086e396f447d4b799ab78be89509be650d6ffbe889" gracePeriod=30
Nov 01 09:16:55.654347 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:55.654309    4931 generic.go:356] "Generic (PLEG): container finished" podID="7241d569-f842-40c9-b66f-78a784161a71" containerID="87941460a892022579080c086e396f447d4b799ab78be89509be650d6ffbe889" exitCode=0
Nov 01 09:16:55.654599 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:55.654575    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq" event={"ID":"7241d569-f842-40c9-b66f-78a784161a71","Type":"ContainerDied","Data":"87941460a892022579080c086e396f447d4b799ab78be89509be650d6ffbe889"}
Nov 01 09:16:55.936990 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:55.936966    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:16:56.118933 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.118898    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7241d569-f842-40c9-b66f-78a784161a71-kube-api-access-27xc4\" (UniqueName: \"kubernetes.io/projected/7241d569-f842-40c9-b66f-78a784161a71-kube-api-access-27xc4\") pod \"7241d569-f842-40c9-b66f-78a784161a71\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") "
Nov 01 09:16:56.119164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.118988    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7241d569-f842-40c9-b66f-78a784161a71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7241d569-f842-40c9-b66f-78a784161a71-openshift-service-ca\") pod \"7241d569-f842-40c9-b66f-78a784161a71\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") "
Nov 01 09:16:56.119164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.119042    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls\") pod \"7241d569-f842-40c9-b66f-78a784161a71\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") "
Nov 01 09:16:56.119164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.119089    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-metrics-certs\") pod \"7241d569-f842-40c9-b66f-78a784161a71\" (UID: \"7241d569-f842-40c9-b66f-78a784161a71\") "
Nov 01 09:16:56.119850 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.119827    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7241d569-f842-40c9-b66f-78a784161a71-openshift-service-ca" pod "7241d569-f842-40c9-b66f-78a784161a71" (UID: "7241d569-f842-40c9-b66f-78a784161a71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:16:56.121570 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.121515    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-metrics-certs" pod "7241d569-f842-40c9-b66f-78a784161a71" (UID: "7241d569-f842-40c9-b66f-78a784161a71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:16:56.122041 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.121998    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls" pod "7241d569-f842-40c9-b66f-78a784161a71" (UID: "7241d569-f842-40c9-b66f-78a784161a71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:16:56.122168 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.122153    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7241d569-f842-40c9-b66f-78a784161a71-kube-api-access-27xc4" pod "7241d569-f842-40c9-b66f-78a784161a71" (UID: "7241d569-f842-40c9-b66f-78a784161a71"). InnerVolumeSpecName "kube-api-access-27xc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:16:56.220422 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.220379    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:56.220422 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.220414    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7241d569-f842-40c9-b66f-78a784161a71-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:56.220422 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.220429    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27xc4\" (UniqueName: \"kubernetes.io/projected/7241d569-f842-40c9-b66f-78a784161a71-kube-api-access-27xc4\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:56.220742 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.220444    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7241d569-f842-40c9-b66f-78a784161a71-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:16:56.408842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.408761    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 09:16:56.409275 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.408841    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 09:16:56.695011 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.694900    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq" event={"ID":"7241d569-f842-40c9-b66f-78a784161a71","Type":"ContainerDied","Data":"66e633a053645953b55dac54069ad9b1a825261f852deec948b05c42cfdeffa8"}
Nov 01 09:16:56.695011 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.694985    4931 scope.go:122] "RemoveContainer" containerID="87941460a892022579080c086e396f447d4b799ab78be89509be650d6ffbe889"
Nov 01 09:16:56.695483 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.695462    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"
Nov 01 09:16:56.853762 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.853729    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"]
Nov 01 09:16:56.856279 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:56.856258    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-28efa1-8559779f45-qrxmq"]
Nov 01 09:16:57.408696 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:57.408657    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 09:16:57.408932 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:57.408737    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 09:16:58.048260 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:58.048225    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7241d569-f842-40c9-b66f-78a784161a71" path="/var/lib/kubelet/pods/7241d569-f842-40c9-b66f-78a784161a71/volumes"
Nov 01 09:16:58.408983 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:58.408944    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 09:16:58.409406 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:58.409005    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 09:16:59.408604 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:59.408560    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 09:16:59.408823 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:16:59.408628    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 09:17:00.708051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:00.707348    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:17:00.709145 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:00.709113    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:17:00.709446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:00.709428    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:17:01.044166 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:17:01.044071    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:17:10.532646 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:10.532617    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:17:10.536696 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:10.536671    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:17:10.562607 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:10.562584    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:17:10.566829 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:10.566810    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:17:22.299292 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.299256    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"]
Nov 01 09:17:22.299830 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.299813    4931 cpu_manager.go:414] "containerMap: removing container" podUID="7241d569-f842-40c9-b66f-78a784161a71" containerName="maas-api"
Nov 01 09:17:22.299895 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.299830    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241d569-f842-40c9-b66f-78a784161a71" containerName="maas-api"
Nov 01 09:17:22.299950 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.299941    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241d569-f842-40c9-b66f-78a784161a71" containerName="maas-api"
Nov 01 09:17:22.304731 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.304709    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309334 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309301    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309348    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309453    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309601 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309518    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309601 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309570    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309642    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309731 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309698    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309780 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309758    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.309837 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.309805    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xc6x\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-kube-api-access-9xc6x\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.407290 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.407260    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"]
Nov 01 09:17:22.411616 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411580    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.411755 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411662    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.411755 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411695    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xc6x\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-kube-api-access-9xc6x\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.411874 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411815    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.411874 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411846    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.411992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411895    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.411992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411925    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.411992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411977    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.412207 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.411989    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.412207 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.412011    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.412207 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.412092    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.412423 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.412398    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.412499 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.412476    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.413518 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.413492    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.414691 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.414664    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.414806 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.414758    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.471578 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.471512    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.471750 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.471638    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xc6x\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-kube-api-access-9xc6x\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:22.621798 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:22.621765    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:23.048190 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:23.048157    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"]
Nov 01 09:17:23.048711 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:17:23.048689    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c47ab33_e553_4f5c_85c8_7c806afc42e2.slice/crio-f54986678e3c8a9a9ce276355f1aab34fb1b6aecd7699ede94287b4553f9fb17 WatchSource:0}: Error finding container f54986678e3c8a9a9ce276355f1aab34fb1b6aecd7699ede94287b4553f9fb17: Status 404 returned error can't find the container with id f54986678e3c8a9a9ce276355f1aab34fb1b6aecd7699ede94287b4553f9fb17
Nov 01 09:17:23.051176 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:23.051147    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:17:23.051250 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:23.051215    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:17:23.051250 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:23.051242    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:17:23.930365 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:23.930330    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" event={"ID":"9c47ab33-e553-4f5c-85c8-7c806afc42e2","Type":"ContainerStarted","Data":"67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1"}
Nov 01 09:17:23.930804 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:23.930381    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" event={"ID":"9c47ab33-e553-4f5c-85c8-7c806afc42e2","Type":"ContainerStarted","Data":"f54986678e3c8a9a9ce276355f1aab34fb1b6aecd7699ede94287b4553f9fb17"}
Nov 01 09:17:24.142562 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:24.142509    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podStartSLOduration=2.142494576 podStartE2EDuration="2.142494576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:24.093772244 +0000 UTC m=+2714.208645228" watchObservedRunningTime="2026-11-01 09:17:24.142494576 +0000 UTC m=+2714.257367543"
Nov 01 09:17:24.623217 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:24.623127    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:24.624114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:24.624088    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 01 09:17:24.624235 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:24.624137    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 01 09:17:25.622963 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:25.622922    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 01 09:17:25.623385 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:25.622986    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 01 09:17:26.623198 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:26.623158    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 01 09:17:26.623601 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:26.623230    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 01 09:17:27.622988 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:27.622949    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 01 09:17:27.623208 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:27.623010    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 01 09:17:28.622588 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:28.622542    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 01 09:17:28.623123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:28.622621    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 01 09:17:29.623086 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:29.623050    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 01 09:17:29.623569 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:29.623108    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 01 09:17:31.027925 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:17:31.027882    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:17:31.058819 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:31.058786    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:31.059474 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:31.059451    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:17:31.060267 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:17:31.060248    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:18:01.025526 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:18:01.025429    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:18:07.795093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.795056    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"]
Nov 01 09:18:07.799415 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.799388    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.880684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.878971    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"]
Nov 01 09:18:07.900393 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900348    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.900571 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900490    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.900571 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900555    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqj4v\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-kube-api-access-hqj4v\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.900685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900585    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.900731 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900673    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.900772 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900729    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.900820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900769    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.900892 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900875    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:07.900987 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:07.900971    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.001866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.001832    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.001866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.001871    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002162 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.001909    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002162 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.001956    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002162 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002057    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002162 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002130    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002425 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002390    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002511 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002474    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002565 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002538    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002620 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002606    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002659    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqj4v\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-kube-api-access-hqj4v\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002675    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.002759 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.002722    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.003094 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.003068    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.005159 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.005132    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.005255 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.005165    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.061205 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.061115    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.061597 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.061574    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqj4v\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-kube-api-access-hqj4v\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.116775 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.116724    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:08.549405 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.549356    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"]
Nov 01 09:18:08.549554 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:18:08.549483    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee58f10_6561_41fe_bcc2_d25018b3ce66.slice/crio-998e938081d992b9f95c5e918c3fc7d95cbcc3ea5334d13e0cff316f8d000502 WatchSource:0}: Error finding container 998e938081d992b9f95c5e918c3fc7d95cbcc3ea5334d13e0cff316f8d000502: Status 404 returned error can't find the container with id 998e938081d992b9f95c5e918c3fc7d95cbcc3ea5334d13e0cff316f8d000502
Nov 01 09:18:08.552364 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.552332    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:08.552483 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.552411    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:08.552483 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:08.552455    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:09.321727 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:09.321683    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" event={"ID":"7ee58f10-6561-41fe-bcc2-d25018b3ce66","Type":"ContainerStarted","Data":"a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef"}
Nov 01 09:18:09.322159 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:09.321747    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" event={"ID":"7ee58f10-6561-41fe-bcc2-d25018b3ce66","Type":"ContainerStarted","Data":"998e938081d992b9f95c5e918c3fc7d95cbcc3ea5334d13e0cff316f8d000502"}
Nov 01 09:18:09.445048 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:09.441223    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podStartSLOduration=2.441201816 podStartE2EDuration="2.441201816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:09.437715729 +0000 UTC m=+2759.552588697" watchObservedRunningTime="2026-11-01 09:18:09.441201816 +0000 UTC m=+2759.556074785"
Nov 01 09:18:10.117611 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:10.117579    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:10.118949 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:10.118923    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 09:18:10.119126 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:10.118978    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 09:18:11.117139 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:11.117104    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 09:18:11.117652 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:11.117154    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 09:18:12.117751 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:12.117716    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 09:18:12.118180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:12.117774    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 09:18:13.118073 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:13.117959    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 09:18:13.118073 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:13.118056    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 09:18:14.141061 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:14.140445    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 09:18:14.141061 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:14.140515    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 09:18:14.844052 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:14.843707    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k" podUID="bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded"
Nov 01 09:18:14.873051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:14.872132    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:15.119049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:15.117384    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 09:18:15.119049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:15.117448    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 09:18:15.211644 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:15.211604    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:16.118069 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:16.118007    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 09:18:16.118252 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:16.118104    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 09:18:18.158193 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:18.158158    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:18:18.158605 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:18.158222    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:18.683698 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:18.683665    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:18.684106 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:18.684078    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:18.771838 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:18.771803    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:18:22.838076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.838040    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"]
Nov 01 09:18:22.842946 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.842924    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921283 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921253    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921460 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921294    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921460 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921374    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgpb\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-kube-api-access-qqgpb\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921537 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921472    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d601123f-a55b-49a6-8271-2b9877866450-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921537 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921506    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d601123f-a55b-49a6-8271-2b9877866450-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921643 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921535    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921698 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921677    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921789 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921745    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.921842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.921797    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:22.945480 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:22.945451    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"]
Nov 01 09:18:23.023196 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023164    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d601123f-a55b-49a6-8271-2b9877866450-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023393 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023213    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d601123f-a55b-49a6-8271-2b9877866450-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023393 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023325    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023517 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023393    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023517 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023478    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023517 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023505    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023682 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023540    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023682 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023574    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023682 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023613    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgpb\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-kube-api-access-qqgpb\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023965 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023770    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.023965 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.023900    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.024111 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.024061    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d601123f-a55b-49a6-8271-2b9877866450-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.024410 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.024376    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.024553 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.024535    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.026778 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.026750    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.026899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.026880    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d601123f-a55b-49a6-8271-2b9877866450-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.083082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.083046    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.083285 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.083252    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgpb\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-kube-api-access-qqgpb\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.158271 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.158234    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:23.600748 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.600713    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"]
Nov 01 09:18:23.600942 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:18:23.600914    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd601123f_a55b_49a6_8271_2b9877866450.slice/crio-1d0aa20bf0c231c13cb68601b3f07f5cc202dd13b3773f331d828c5d03de038c WatchSource:0}: Error finding container 1d0aa20bf0c231c13cb68601b3f07f5cc202dd13b3773f331d828c5d03de038c: Status 404 returned error can't find the container with id 1d0aa20bf0c231c13cb68601b3f07f5cc202dd13b3773f331d828c5d03de038c
Nov 01 09:18:23.603484 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.603453    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:23.603558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.603523    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:23.603558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:23.603554    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:24.446212 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:24.446174    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" event={"ID":"d601123f-a55b-49a6-8271-2b9877866450","Type":"ContainerStarted","Data":"c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83"}
Nov 01 09:18:24.446609 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:24.446240    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" event={"ID":"d601123f-a55b-49a6-8271-2b9877866450","Type":"ContainerStarted","Data":"1d0aa20bf0c231c13cb68601b3f07f5cc202dd13b3773f331d828c5d03de038c"}
Nov 01 09:18:24.658935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:24.658868    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podStartSLOduration=2.658850358 podStartE2EDuration="2.658850358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:24.609761068 +0000 UTC m=+2774.724634033" watchObservedRunningTime="2026-11-01 09:18:24.658850358 +0000 UTC m=+2774.773723328"
Nov 01 09:18:25.165050 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:25.163368    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:25.183054 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:25.171146    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 09:18:25.183054 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:25.171211    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 09:18:26.158934 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:26.158896    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 09:18:26.159493 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:26.158972    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 09:18:27.158591 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:27.158557    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 09:18:27.158803 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:27.158631    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 09:18:28.163184 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:28.163143    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 09:18:28.163796 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:28.163763    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 09:18:29.158683 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:29.158652    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 09:18:29.158912 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:29.158706    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 09:18:30.159010 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:30.158975    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 09:18:30.159478 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:30.159087    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 09:18:31.030084 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:18:31.030041    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:18:31.701911 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:31.701871    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:31.702738 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:31.702343    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:31.703634 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:31.703614    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:18:38.019103 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.019061    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"]
Nov 01 09:18:38.023187 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.023167    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.130038 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.129986    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.130240 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.130166    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.130240 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.130219    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.130362 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.130347    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3909073-3752-417c-a8ee-bb847881767a-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.131234 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.131200    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3909073-3752-417c-a8ee-bb847881767a-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.142466 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.131443    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.142466 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.131579    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.142466 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.131654    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txvt\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-kube-api-access-2txvt\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.142466 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.132275    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.222218 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.222180    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"]
Nov 01 09:18:38.233529 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.233493    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.233701 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.233557    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.233701 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.233617    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txvt\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-kube-api-access-2txvt\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.233789 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.233702    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.233845 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.233828    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.233903 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.233880    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.233957 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.233920    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.234008 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.233962    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3909073-3752-417c-a8ee-bb847881767a-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.234119 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.234013    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3909073-3752-417c-a8ee-bb847881767a-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.234311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.234285    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.234394 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.234371    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.234662 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.234643    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.234775 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.234757    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.234848 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.234792    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3909073-3752-417c-a8ee-bb847881767a-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.236634 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.236608    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.237333 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.237311    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3909073-3752-417c-a8ee-bb847881767a-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.292908 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.292829    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.293082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.292977    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txvt\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-kube-api-access-2txvt\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.337343 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.337313    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:38.769219 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:18:38.769187    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3909073_3752_417c_a8ee_bb847881767a.slice/crio-8f87a6d1762585ab242c0cb4c3801cedff4769d4bf36c40adfda176934aa0beb WatchSource:0}: Error finding container 8f87a6d1762585ab242c0cb4c3801cedff4769d4bf36c40adfda176934aa0beb: Status 404 returned error can't find the container with id 8f87a6d1762585ab242c0cb4c3801cedff4769d4bf36c40adfda176934aa0beb
Nov 01 09:18:38.770884 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.770857    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"]
Nov 01 09:18:38.772067 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.772007    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:38.772162 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.772118    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:38.772230 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:38.772169    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:18:39.599871 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:39.599328    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" event={"ID":"b3909073-3752-417c-a8ee-bb847881767a","Type":"ContainerStarted","Data":"8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce"}
Nov 01 09:18:39.599871 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:39.599400    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" event={"ID":"b3909073-3752-417c-a8ee-bb847881767a","Type":"ContainerStarted","Data":"8f87a6d1762585ab242c0cb4c3801cedff4769d4bf36c40adfda176934aa0beb"}
Nov 01 09:18:39.775052 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:39.766317    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podStartSLOduration=2.766295641 podStartE2EDuration="2.766295641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:39.76531111 +0000 UTC m=+2789.880184078" watchObservedRunningTime="2026-11-01 09:18:39.766295641 +0000 UTC m=+2789.881168622"
Nov 01 09:18:40.337673 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:40.337641    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:40.339387 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:40.339357    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:40.339523 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:40.339408    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:41.337675 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:41.337623    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:41.338300 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:41.337694    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:42.343044 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:42.342979    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:42.343687 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:42.343653    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:43.338457 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:43.338416    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:43.338760 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:43.338729    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:43.962170 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:43.962122    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.185049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.184138    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded"
Nov 01 09:18:44.241126 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.238110    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.264177 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.264125    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.331322 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.331268    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.340482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.340448    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:44.340797 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.340749    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:44.562390 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.562257    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.588954 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.588136    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:44.642052 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:44.641133    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.197052 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.195125    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.197052 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.195574    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 09:18:45.269051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.265343    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 09:18:45.269051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.266093    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 09:18:45.332079 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.331988    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.333218 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.332379    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 09:18:45.353093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.349414    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:45.353093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.349476    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:45.655160 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.609785    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:45.655160 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.610275    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 09:18:45.893055 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:45.890943    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.082110 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.081123    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.208052 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.206557    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.287145 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.287089    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.339625 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.339532    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:46.339881 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.339851    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:46.385194 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.385145    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.614297 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.614231    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.774049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.761679    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.774049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.769062    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.920108 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.894120    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 09:18:46.920108 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.894666    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 09:18:46.954932 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.952465    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k" podUID="bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:46.961079 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:46.955993    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 09:18:47.083594 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.083161    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:47.138087 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.132138    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 09:18:47.142287 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.138943    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:18:47.142287 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.138999    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:47.224994 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.224882    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:47.283558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.283515    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 09:18:47.288978 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.288214    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:47.312460 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.311633    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 09:18:47.344325 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.344293    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:47.344594 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.344572    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:47.367875 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.367845    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 09:18:47.429173 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.429124    4931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:47.533914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.533827    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 09:18:47.662603 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:47.662572    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 09:18:48.338086 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:48.338053    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 01 09:18:48.338505 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:48.338106    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 01 09:18:50.381205 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:50.381165    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:18:50.381706 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:50.381232    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:51.381828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:51.381753    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:18:51.382546 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:51.382508    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:18:51.460312 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:51.460280    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:51.460727 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:51.460701    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:18:51.461982 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:18:51.461944    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:19:01.025768 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:19:01.025729    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:19:12.223072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:12.219240    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:12.346128 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:12.331116    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 09:19:12.425579 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:12.425534    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 01 09:19:12.537614 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:12.537128    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 01 09:19:12.580201 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:12.580142    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:19:31.024266 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:19:31.024168    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:19:42.033730 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.033698    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"]
Nov 01 09:19:42.037730 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.037714    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.114525 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.114473    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqk8s\" (UniqueName: \"kubernetes.io/projected/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-kube-api-access-jqk8s\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.114699 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.114549    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.114743 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.114704    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.114790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.114751    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.137340 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.137309    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-035625-serving-cert\""
Nov 01 09:19:42.140459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.140439    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"]
Nov 01 09:19:42.216094 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.216065    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqk8s\" (UniqueName: \"kubernetes.io/projected/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-kube-api-access-jqk8s\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.216277 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.216105    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.216277 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.216163    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.216277 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.216186    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.216400 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:19:42.216291    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-serving-cert: secret "maas-api-e2e-shared-a-w3-035625-serving-cert" not found
Nov 01 09:19:42.216400 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:19:42.216372    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls podName:cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0 nodeName:}" failed. No retries permitted until 2026-11-01 09:19:42.716352797 +0000 UTC m=+2852.831225761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls") pod "maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md" (UID: "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0") : secret "maas-api-e2e-shared-a-w3-035625-serving-cert" not found
Nov 01 09:19:42.217105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.217004    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.218951 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.218933    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.275817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.275790    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqk8s\" (UniqueName: \"kubernetes.io/projected/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-kube-api-access-jqk8s\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.722238 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.722204    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.724749 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.724726    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:42.952216 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:42.952187    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:43.325743 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:43.325715    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"]
Nov 01 09:19:43.326917 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:19:43.326892    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbfa2ef_f8d6_4b93_8f35_fa120348e6e0.slice/crio-4433a0426a104a6a390c0654ca5ed89d9ccc8cf2b306f7f7c2cd0cf9d6861e4f WatchSource:0}: Error finding container 4433a0426a104a6a390c0654ca5ed89d9ccc8cf2b306f7f7c2cd0cf9d6861e4f: Status 404 returned error can't find the container with id 4433a0426a104a6a390c0654ca5ed89d9ccc8cf2b306f7f7c2cd0cf9d6861e4f
Nov 01 09:19:43.328699 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:43.328682    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:19:43.485125 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:43.485088    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md" event={"ID":"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0","Type":"ContainerStarted","Data":"4433a0426a104a6a390c0654ca5ed89d9ccc8cf2b306f7f7c2cd0cf9d6861e4f"}
Nov 01 09:19:44.493531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:44.493430    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md" event={"ID":"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0","Type":"ContainerStarted","Data":"67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b"}
Nov 01 09:19:44.493967 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:44.493524    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:19:44.610054 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:44.609990    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md" podStartSLOduration=2.722822614 podStartE2EDuration="3.609975881s" totalImagesPullingTime="887.153267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:44.607514004 +0000 UTC m=+2854.722386982" watchObservedRunningTime="2026-11-01 09:19:44.609975881 +0000 UTC m=+2854.724848851"
Nov 01 09:19:50.505118 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:19:50.505085    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:20:01.023914 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:20:01.023861    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:20:31.025951 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:20:31.025911    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:21:01.027415 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:21:01.027333    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:21:31.023373 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:21:31.023337    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:21:53.469623 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:53.469585    4931 generic.go:356] "Generic (PLEG): container finished" podID="5cdcfa74-fd37-405c-8850-47dfa5b238e5" containerID="85a762d1dc056bc0832547d7f67e1dcff99a3a735eba705c0024e3c3476245ee" exitCode=1
Nov 01 09:21:53.470126 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:53.469657    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4" event={"ID":"5cdcfa74-fd37-405c-8850-47dfa5b238e5","Type":"ContainerDied","Data":"85a762d1dc056bc0832547d7f67e1dcff99a3a735eba705c0024e3c3476245ee"}
Nov 01 09:21:53.470188 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:53.470146    4931 scope.go:122] "RemoveContainer" containerID="85a762d1dc056bc0832547d7f67e1dcff99a3a735eba705c0024e3c3476245ee"
Nov 01 09:21:53.471751 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:53.471731    4931 generic.go:356] "Generic (PLEG): container finished" podID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerID="3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a" exitCode=0
Nov 01 09:21:53.471861 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:53.471771    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp" event={"ID":"76ef7b97-d4b7-4482-9026-98a0ffffeaf1","Type":"ContainerDied","Data":"3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a"}
Nov 01 09:21:53.472108 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:53.472093    4931 scope.go:122] "RemoveContainer" containerID="3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a"
Nov 01 09:21:53.520496 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:53.520468    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 01 09:21:54.480353 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:54.480316    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp" event={"ID":"76ef7b97-d4b7-4482-9026-98a0ffffeaf1","Type":"ContainerStarted","Data":"c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9"}
Nov 01 09:21:54.480810 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:54.480532    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 09:21:54.482521 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:54.482488    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4" event={"ID":"5cdcfa74-fd37-405c-8850-47dfa5b238e5","Type":"ContainerStarted","Data":"757179ab4c1116d86cc214056a2000725dbfd4d5f72925ba9152d6c26b9f4f74"}
Nov 01 09:21:54.482700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:21:54.482685    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 09:22:01.023868 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:22:01.023828    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:22:05.492230 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:22:05.492203    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zslv4"
Nov 01 09:22:05.492658 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:22:05.492268    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 09:22:10.579680 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:22:10.579651    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:22:10.583674 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:22:10.583647    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:22:10.610759 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:22:10.610736    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:22:10.614450 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:22:10.614433    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:22:31.025628 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:22:31.025543    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:23:01.023563 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:23:01.023521    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:23:08.975615 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:08.975578    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"]
Nov 01 09:23:08.980255 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:08.980227    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.038764 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.038729    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"]
Nov 01 09:23:09.077954 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.077923    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-035625\"/\"openshift-service-ca.crt\""
Nov 01 09:23:09.078222 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.078170    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-035625\"/\"default-dockercfg-mmrx2\""
Nov 01 09:23:09.078222 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.078174    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-035625\"/\"auth-test-model-035625-kserve-self-signed-certs\""
Nov 01 09:23:09.080442 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.080428    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-035625\"/\"kube-root-ca.crt\""
Nov 01 09:23:09.108098 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.108070    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-model-cache\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.108232 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.108109    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8958de78-bacf-497f-8051-f949459e57a9-tls-certs\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.108300 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.108283    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-home\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.108397 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.108384    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-tmp-dir\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.108434 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.108409    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh7pw\" (UniqueName: \"kubernetes.io/projected/8958de78-bacf-497f-8051-f949459e57a9-kube-api-access-sh7pw\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.108486 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.108447    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-dshm\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.209840 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.209801    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-home\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.210092 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.209903    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-tmp-dir\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.210092 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.209951    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh7pw\" (UniqueName: \"kubernetes.io/projected/8958de78-bacf-497f-8051-f949459e57a9-kube-api-access-sh7pw\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.210092 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.210053    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-dshm\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.210331 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.210306    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-home\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.210402 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.210364    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-tmp-dir\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.210491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.210476    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-model-cache\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.210549 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.210516    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8958de78-bacf-497f-8051-f949459e57a9-tls-certs\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.210764 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.210742    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-model-cache\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.213419 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.213393    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-dshm\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.213922 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.213905    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8958de78-bacf-497f-8051-f949459e57a9-tls-certs\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.269972 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.269886    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh7pw\" (UniqueName: \"kubernetes.io/projected/8958de78-bacf-497f-8051-f949459e57a9-kube-api-access-sh7pw\") pod \"auth-test-model-035625-kserve-6f8f58c95c-f4nnj\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") " pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.294878 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.294838    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:09.685773 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:09.685742    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"]
Nov 01 09:23:09.686811 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:23:09.686779    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8958de78_bacf_497f_8051_f949459e57a9.slice/crio-accc624004265c1e49cf552118d2958f818836d1154ac2436121299f22fadc29 WatchSource:0}: Error finding container accc624004265c1e49cf552118d2958f818836d1154ac2436121299f22fadc29: Status 404 returned error can't find the container with id accc624004265c1e49cf552118d2958f818836d1154ac2436121299f22fadc29
Nov 01 09:23:10.050715 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:10.050638    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj" event={"ID":"8958de78-bacf-497f-8051-f949459e57a9","Type":"ContainerStarted","Data":"cb17de9c562c7ad2400c4d0a3530dee8a4f4153dd8a3b11985ac63d89786d5d9"}
Nov 01 09:23:10.050715 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:10.050678    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj" event={"ID":"8958de78-bacf-497f-8051-f949459e57a9","Type":"ContainerStarted","Data":"accc624004265c1e49cf552118d2958f818836d1154ac2436121299f22fadc29"}
Nov 01 09:23:10.182467 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:10.182400    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj" podStartSLOduration=2.182378732 podStartE2EDuration="2.182378732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:23:10.181664021 +0000 UTC m=+3060.296536989" watchObservedRunningTime="2026-11-01 09:23:10.182378732 +0000 UTC m=+3060.297251702"
Nov 01 09:23:10.295446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:10.295412    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:19.296003 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:19.295963    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:19.309119 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:19.309085    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:20.136595 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:23:20.136562    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:23:31.024016 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:23:31.023973    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:24:01.023586 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:24:01.023503    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:24:10.596093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.596051    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"]
Nov 01 09:24:10.600498 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.600476    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.654367 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.654327    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"]
Nov 01 09:24:10.695856 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.695822    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e91af4-7ae5-487f-9244-80238b88034a-tls-certs\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.696070 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.695952    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-tmp-dir\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.696070 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.696017    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-model-cache\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.696198 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.696100    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wslpl\" (UniqueName: \"kubernetes.io/projected/b6e91af4-7ae5-487f-9244-80238b88034a-kube-api-access-wslpl\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.696262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.696192    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-home\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.696339 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.696322    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-dshm\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.697084 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.697061    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"kube-root-ca.crt\""
Nov 01 09:24:10.697202 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.697061    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"openshift-service-ca.crt\""
Nov 01 09:24:10.698231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.698209    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"default-dockercfg-6tkpr\""
Nov 01 09:24:10.698231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.698210    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a87a1d\"/\"auth-test-model-a87a1d-kserve-self-signed-certs\""
Nov 01 09:24:10.798582 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.798528    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e91af4-7ae5-487f-9244-80238b88034a-tls-certs\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.798790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.798673    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-tmp-dir\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.798790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.798768    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-model-cache\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.798904 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.798804    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslpl\" (UniqueName: \"kubernetes.io/projected/b6e91af4-7ae5-487f-9244-80238b88034a-kube-api-access-wslpl\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.798904 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.798865    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-home\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.799010 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.798927    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-dshm\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.799683 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.799604    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-model-cache\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.801223 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.801197    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-home\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.801507 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.801484    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-tmp-dir\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.808664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.808629    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e91af4-7ae5-487f-9244-80238b88034a-tls-certs\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.816548 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.816519    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-dshm\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.859751 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.859714    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslpl\" (UniqueName: \"kubernetes.io/projected/b6e91af4-7ae5-487f-9244-80238b88034a-kube-api-access-wslpl\") pod \"auth-test-model-a87a1d-kserve-5dcf84c769-cgs86\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") " pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:10.914843 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:10.914807    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:11.334129 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:11.334098    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"]
Nov 01 09:24:11.336971 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:24:11.336939    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6e91af4_7ae5_487f_9244_80238b88034a.slice/crio-801a4cd26c512f63df6691353e0714550a60e6661520353a4d7780b7b7b12192 WatchSource:0}: Error finding container 801a4cd26c512f63df6691353e0714550a60e6661520353a4d7780b7b7b12192: Status 404 returned error can't find the container with id 801a4cd26c512f63df6691353e0714550a60e6661520353a4d7780b7b7b12192
Nov 01 09:24:11.529799 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:11.529755    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86" event={"ID":"b6e91af4-7ae5-487f-9244-80238b88034a","Type":"ContainerStarted","Data":"ecec6a084686a23ad0496674bf8026cde1678d080139055769a61c14573d6526"}
Nov 01 09:24:11.529799 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:11.529806    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86" event={"ID":"b6e91af4-7ae5-487f-9244-80238b88034a","Type":"ContainerStarted","Data":"801a4cd26c512f63df6691353e0714550a60e6661520353a4d7780b7b7b12192"}
Nov 01 09:24:11.645949 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:11.645889    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86" podStartSLOduration=1.645871998 podStartE2EDuration="1.645871998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:24:11.641690401 +0000 UTC m=+3121.756563370" watchObservedRunningTime="2026-11-01 09:24:11.645871998 +0000 UTC m=+3121.760744967"
Nov 01 09:24:11.915472 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:11.915388    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:20.915093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:20.915053    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:20.928828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:20.928804    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:21.610417 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:21.610381    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:24:31.031293 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:24:31.031245    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:24:44.334170 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.334131    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"]
Nov 01 09:24:44.337882 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.337859    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.436416 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.436380    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5b96ac\"/\"kube-root-ca.crt\""
Nov 01 09:24:44.436642 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.436624    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5b96ac\"/\"openshift-service-ca.crt\""
Nov 01 09:24:44.436693 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.436661    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5b96ac\"/\"auth-test-model-5b96ac-kserve-self-signed-certs\""
Nov 01 09:24:44.436734 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.436708    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5b96ac\"/\"default-dockercfg-j4mbh\""
Nov 01 09:24:44.440193 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.440174    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"]
Nov 01 09:24:44.453447 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.453424    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tls-certs\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.453598 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.453454    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tmp-dir\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.453655 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.453598    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-model-cache\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.453655 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.453639    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpvxx\" (UniqueName: \"kubernetes.io/projected/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-kube-api-access-xpvxx\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.453773 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.453759    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-home\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.453809 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.453785    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-dshm\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555094 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555055    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tls-certs\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555106    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tmp-dir\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555205    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-model-cache\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555311 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555232    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpvxx\" (UniqueName: \"kubernetes.io/projected/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-kube-api-access-xpvxx\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555483 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555319    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-home\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555483 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555354    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-dshm\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555632 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555608    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tmp-dir\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555696 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555622    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-model-cache\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.555793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.555769    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-home\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.557678 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.557652    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-dshm\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.557931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.557911    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tls-certs\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.615854 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.615826    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpvxx\" (UniqueName: \"kubernetes.io/projected/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-kube-api-access-xpvxx\") pod \"auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") " pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:44.655905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:44.655867    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:45.041240 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:24:45.041204    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod824c6bdd_5ccc_4e8b_89b7_b23d901b36f0.slice/crio-0789c73e3e1b2537fa1bda49ea2523fdaabb6cf0384af29f4502d2c9477fecc1 WatchSource:0}: Error finding container 0789c73e3e1b2537fa1bda49ea2523fdaabb6cf0384af29f4502d2c9477fecc1: Status 404 returned error can't find the container with id 0789c73e3e1b2537fa1bda49ea2523fdaabb6cf0384af29f4502d2c9477fecc1
Nov 01 09:24:45.042946 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:45.042912    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"]
Nov 01 09:24:45.780453 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:45.780417    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k" event={"ID":"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0","Type":"ContainerStarted","Data":"19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138"}
Nov 01 09:24:45.780957 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:45.780469    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k" event={"ID":"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0","Type":"ContainerStarted","Data":"0789c73e3e1b2537fa1bda49ea2523fdaabb6cf0384af29f4502d2c9477fecc1"}
Nov 01 09:24:45.899487 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:45.899417    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k" podStartSLOduration=1.899397553 podStartE2EDuration="1.899397553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:24:45.894088423 +0000 UTC m=+3156.008961392" watchObservedRunningTime="2026-11-01 09:24:45.899397553 +0000 UTC m=+3156.014270522"
Nov 01 09:24:46.656424 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:46.656390    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:54.656946 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:54.656906    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:54.674726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:54.674696    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:24:54.861365 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:24:54.861330    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:25:01.026442 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:25:01.026400    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:25:31.036350 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:25:31.035886    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:26:01.029722 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:26:01.029662    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:26:30.754191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.754153    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"]
Nov 01 09:26:30.760133 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.760096    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.768517 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.768476    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.768689 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.768526    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.768689 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.768656    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.768817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.768763    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.768817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.768802    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872aebee-a5d5-4a7e-a884-7cf86ab02777-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.768931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.768887    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.768991 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.768953    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxxh\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-kube-api-access-pnxxh\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.769108 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.768994    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.769201 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.769185    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.863885 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.863843    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"]
Nov 01 09:26:30.869899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.869865    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.869925    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.869953    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.869983    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870085    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870113    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872aebee-a5d5-4a7e-a884-7cf86ab02777-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870163    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870217    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxxh\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-kube-api-access-pnxxh\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870255    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870463 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870344    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870674 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870574    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870674 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870632    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.870913 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.870864    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.871739 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.871714    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872aebee-a5d5-4a7e-a884-7cf86ab02777-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.875416 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.875387    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.875652 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.875630    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.894231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.894202    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a83714-openshift-default-dockercfg-6fhhb\""
Nov 01 09:26:30.930862 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.930834    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:30.931067 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:30.930900    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxxh\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-kube-api-access-pnxxh\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:31.024349 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:26:31.024261    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:26:31.078615 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:31.078581    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:31.504274 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:31.504243    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"]
Nov 01 09:26:31.504948 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:26:31.504917    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872aebee_a5d5_4a7e_a884_7cf86ab02777.slice/crio-6395b80b21cf1e98deef7817a78210610e53bbe047e670b7e12afc709b63f2ab WatchSource:0}: Error finding container 6395b80b21cf1e98deef7817a78210610e53bbe047e670b7e12afc709b63f2ab: Status 404 returned error can't find the container with id 6395b80b21cf1e98deef7817a78210610e53bbe047e670b7e12afc709b63f2ab
Nov 01 09:26:31.507835 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:31.507803    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:26:31.507910 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:31.507881    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:26:31.507944 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:31.507914    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:26:31.538481 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:31.538447    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" event={"ID":"872aebee-a5d5-4a7e-a884-7cf86ab02777","Type":"ContainerStarted","Data":"6395b80b21cf1e98deef7817a78210610e53bbe047e670b7e12afc709b63f2ab"}
Nov 01 09:26:32.548577 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:32.548544    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" event={"ID":"872aebee-a5d5-4a7e-a884-7cf86ab02777","Type":"ContainerStarted","Data":"3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b"}
Nov 01 09:26:32.717700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:32.717626    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" podStartSLOduration=2.7176047629999998 podStartE2EDuration="2.717604763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:26:32.711976169 +0000 UTC m=+3262.826849144" watchObservedRunningTime="2026-11-01 09:26:32.717604763 +0000 UTC m=+3262.832477741"
Nov 01 09:26:33.085147 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:33.084369    4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 09:26:33.085147 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:33.084433    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 09:26:33.085147 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:33.084385    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:34.083459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:34.083418    4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 09:26:34.084120 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:34.084088    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 09:26:35.086055 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:35.082353    4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 09:26:35.086055 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:35.082416    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 09:26:36.084215 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:36.084158    4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 09:26:36.084437 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:36.084242    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 09:26:37.080168 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:37.080130    4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 09:26:37.080765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:37.080732    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 09:26:38.085315 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:38.085285    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:38.085784 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:38.085763    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:26:38.086620 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:26:38.086601    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:27:01.024861 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:27:01.024824    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:27:10.644519 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:10.644489    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:27:10.648923 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:10.648885    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:27:10.673734 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:10.673707    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:27:10.678646 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:10.678619    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:27:31.025257 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:27:31.025212    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:27:53.900124 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:53.900088    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"]
Nov 01 09:27:53.900721 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:53.900298    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy" containerID="cri-o://31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6" gracePeriod=30
Nov 01 09:27:57.145078 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:57.145038    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"]
Nov 01 09:27:57.145542 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:57.145267    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy" containerID="cri-o://c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83" gracePeriod=30
Nov 01 09:27:57.192978 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:57.192946    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"]
Nov 01 09:27:57.193270 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:57.193217    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" containerID="cri-o://67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1" gracePeriod=30
Nov 01 09:27:57.348733 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:57.348694    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"]
Nov 01 09:27:57.348973 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:57.348948    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy" containerID="cri-o://2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d" gracePeriod=30
Nov 01 09:27:57.352404 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:57.352374    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"]
Nov 01 09:27:57.352643 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:57.352600    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy" containerID="cri-o://8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce" gracePeriod=30
Nov 01 09:27:59.189218 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.189192    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 09:27:59.201612 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.201581    4931 generic.go:356] "Generic (PLEG): container finished" podID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerID="31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6" exitCode=0
Nov 01 09:27:59.201781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.201641    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"
Nov 01 09:27:59.201781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.201659    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" event={"ID":"91aedd00-ab73-46a1-8ef7-11d6440a4706","Type":"ContainerDied","Data":"31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6"}
Nov 01 09:27:59.201781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.201712    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg" event={"ID":"91aedd00-ab73-46a1-8ef7-11d6440a4706","Type":"ContainerDied","Data":"2ec94dfc48a98b87877a6066cccd8fec1852fe4a7412ee25eb38a728d1179097"}
Nov 01 09:27:59.201781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.201734    4931 scope.go:122] "RemoveContainer" containerID="31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6"
Nov 01 09:27:59.212556 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.212534    4931 scope.go:122] "RemoveContainer" containerID="78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0"
Nov 01 09:27:59.222364 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.222345    4931 scope.go:122] "RemoveContainer" containerID="31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6"
Nov 01 09:27:59.222647 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:27:59.222624    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6\": container with ID starting with 31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6 not found: ID does not exist" containerID="31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6"
Nov 01 09:27:59.222701 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.222663    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6"} err="failed to get container status \"31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6\": rpc error: code = NotFound desc = could not find container \"31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6\": container with ID starting with 31f8f08c44ab5b7d16e38c5399847c9e21d7181623bdd12d3d037876216d0ce6 not found: ID does not exist"
Nov 01 09:27:59.222701 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.222688    4931 scope.go:122] "RemoveContainer" containerID="78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0"
Nov 01 09:27:59.223012 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:27:59.222992    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0\": container with ID starting with 78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0 not found: ID does not exist" containerID="78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0"
Nov 01 09:27:59.223102 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.223081    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0"} err="failed to get container status \"78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0\": rpc error: code = NotFound desc = could not find container \"78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0\": container with ID starting with 78668bf27955c187681c7162adddd8aeb01098f9dc1170c2948d29b1d4bf2ae0 not found: ID does not exist"
Nov 01 09:27:59.272735 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.272703    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-token\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-token\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.272735 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.272737    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91aedd00-ab73-46a1-8ef7-11d6440a4706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aedd00-ab73-46a1-8ef7-11d6440a4706-istiod-ca-cert\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.272963 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.272803    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-envoy\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.272963 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.272838    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-credential-socket\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.273520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.272895    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-data\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.273520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.273479    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-kube-api-access-rvqzd\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-kube-api-access-rvqzd\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.273520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.273510    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91aedd00-ab73-46a1-8ef7-11d6440a4706-istiod-ca-cert" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:27:59.273738 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.273524    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-socket\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.273738 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.273613    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-certs\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.273738 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.273660    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-podinfo\") pod \"91aedd00-ab73-46a1-8ef7-11d6440a4706\" (UID: \"91aedd00-ab73-46a1-8ef7-11d6440a4706\") "
Nov 01 09:27:59.273893 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.273873    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-credential-socket" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:59.274266 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.274100    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-socket" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:59.274510 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.274421    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-data" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:59.274510 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.274504    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-certs" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:59.276002 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.275969    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-podinfo" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:27:59.276300 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.276277    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-kube-api-access-rvqzd" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "kube-api-access-rvqzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:27:59.276561 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.276369    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-token" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:27:59.278579 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.278553    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-envoy" pod "91aedd00-ab73-46a1-8ef7-11d6440a4706" (UID: "91aedd00-ab73-46a1-8ef7-11d6440a4706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:27:59.280684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280659    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.280790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280690    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.280790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280713    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvqzd\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-kube-api-access-rvqzd\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.280790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280728    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.280790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280743    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.280790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280757    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.280790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280779    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.288238 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280793    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aedd00-ab73-46a1-8ef7-11d6440a4706-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.288238 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.280810    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aedd00-ab73-46a1-8ef7-11d6440a4706-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:27:59.623934 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.623903    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"]
Nov 01 09:27:59.629069 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:27:59.629041    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-x8wfg"]
Nov 01 09:28:00.049772 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:00.049737    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" path="/var/lib/kubelet/pods/91aedd00-ab73-46a1-8ef7-11d6440a4706/volumes"
Nov 01 09:28:00.716073 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:00.711311    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:28:00.716073 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:00.711426    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:28:01.025060 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:28:01.024965    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:28:01.061726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:01.061691    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:28:01.061948 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:01.061759    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:28:01.705637 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:01.705602    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:28:01.705842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:01.705679    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:28:02.552527 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.552492    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:28:02.573677 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.573645    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:28:02.678150 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.678124    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:28:02.705317 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.705254    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:28:02.735928 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.735895    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istiod-ca-cert\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.735947    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-envoy\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.736151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.735970    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-certs\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736010    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-kube-api-access-qqgpb\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-kube-api-access-qqgpb\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.736151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736128    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-credential-socket\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736386 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736172    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-data\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.736386 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736204    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-data\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736386 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736231    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-credential-socket\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.736386 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736264    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-socket\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736386 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736310    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-kube-api-access-9xc6x\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-kube-api-access-9xc6x\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736386 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736351    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-token\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-token\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736386 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736384    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-podinfo\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736433    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-envoy\") pod \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\" (UID: \"9c47ab33-e553-4f5c-85c8-7c806afc42e2\") "
Nov 01 09:28:02.736741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736465    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istiod-ca-cert" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:28:02.736741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736561    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-istio-token\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-istio-token\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.736741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736651    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-socket" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.736741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736658    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-socket\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.736741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736729    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d601123f-a55b-49a6-8271-2b9877866450-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d601123f-a55b-49a6-8271-2b9877866450-istiod-ca-cert\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.737101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736735    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-data" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.737101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736761    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-certs\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.737101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736832    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-data" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.737101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736839    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d601123f-a55b-49a6-8271-2b9877866450-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d601123f-a55b-49a6-8271-2b9877866450-istio-podinfo\") pod \"d601123f-a55b-49a6-8271-2b9877866450\" (UID: \"d601123f-a55b-49a6-8271-2b9877866450\") "
Nov 01 09:28:02.737101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.736844    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-credential-socket" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.737101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737055    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-credential-socket" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737142    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d601123f-a55b-49a6-8271-2b9877866450-istiod-ca-cert" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737538    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737556    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737573    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737587    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737601    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737617    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737631    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d601123f-a55b-49a6-8271-2b9877866450-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737729    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-socket" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.737914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.737884    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-certs" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.739379 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.739341    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-certs" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.739692 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.739665    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-envoy" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.740790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.740764    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-istio-token" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:02.741273 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.741048    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d601123f-a55b-49a6-8271-2b9877866450-istio-podinfo" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:28:02.741701 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.741673    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-token" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:02.741829 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.741719    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-kube-api-access-qqgpb" pod "d601123f-a55b-49a6-8271-2b9877866450" (UID: "d601123f-a55b-49a6-8271-2b9877866450"). InnerVolumeSpecName "kube-api-access-qqgpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:02.742273 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.742242    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-kube-api-access-9xc6x" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "kube-api-access-9xc6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:02.742767 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.742740    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-envoy" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.743086 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.743002    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-podinfo" pod "9c47ab33-e553-4f5c-85c8-7c806afc42e2" (UID: "9c47ab33-e553-4f5c-85c8-7c806afc42e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:28:02.838621 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.838591    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-envoy\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.838621 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.838631    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-socket\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.838876 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.838662    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-istio-token\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-istio-token\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.838876 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.838686    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6728c86f-880f-4ea6-8755-7bc9c01d4981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6728c86f-880f-4ea6-8755-7bc9c01d4981-istiod-ca-cert\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.838876 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.838748    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-data\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.839088 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.838951    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-socket" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.839088 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839057    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-data" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.839211 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839193    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3909073-3752-417c-a8ee-bb847881767a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3909073-3752-417c-a8ee-bb847881767a-istio-podinfo\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.839278 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839249    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-socket\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.839332 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839287    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6728c86f-880f-4ea6-8755-7bc9c01d4981-istiod-ca-cert" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:28:02.839332 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839302    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-envoy\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.839433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839339    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3909073-3752-417c-a8ee-bb847881767a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3909073-3752-417c-a8ee-bb847881767a-istiod-ca-cert\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.839433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839411    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-token\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-token\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.839534 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839441    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-certs\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.839534 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839516    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-podinfo\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.839638 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839581    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-data\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.839638 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839621    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-credential-socket\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.839738 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839658    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-kube-api-access-qmh8p\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-kube-api-access-qmh8p\") pod \"6728c86f-880f-4ea6-8755-7bc9c01d4981\" (UID: \"6728c86f-880f-4ea6-8755-7bc9c01d4981\") "
Nov 01 09:28:02.839738 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839694    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-credential-socket\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.839846 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839733    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-kube-api-access-2txvt\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-kube-api-access-2txvt\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.839846 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.839823    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-certs\") pod \"b3909073-3752-417c-a8ee-bb847881767a\" (UID: \"b3909073-3752-417c-a8ee-bb847881767a\") "
Nov 01 09:28:02.840236 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840214    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-data" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.840338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840244    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6728c86f-880f-4ea6-8755-7bc9c01d4981-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840266    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840280    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840295    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d601123f-a55b-49a6-8271-2b9877866450-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840308    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840320    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840334    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqgpb\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-kube-api-access-qqgpb\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840349    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xc6x\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-kube-api-access-9xc6x\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840364    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840378    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840391    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c47ab33-e553-4f5c-85c8-7c806afc42e2-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840406    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d601123f-a55b-49a6-8271-2b9877866450-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840422    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840435    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d601123f-a55b-49a6-8271-2b9877866450-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840625    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3909073-3752-417c-a8ee-bb847881767a-istiod-ca-cert" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:28:02.840717 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840627    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-certs" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.844105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.840795    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-credential-socket" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.844105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.841266    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-socket" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.844105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.841492    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-credential-socket" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.844105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.842125    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-certs" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.844849 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.844822    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3909073-3752-417c-a8ee-bb847881767a-istio-podinfo" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:28:02.845249 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.845226    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-envoy" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.845592 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.845566    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-kube-api-access-2txvt" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "kube-api-access-2txvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:02.846529 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.846494    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-podinfo" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:28:02.854307 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.854255    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-envoy" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:28:02.854468 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.854445    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-token" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:02.854639 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.854616    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-istio-token" pod "b3909073-3752-417c-a8ee-bb847881767a" (UID: "b3909073-3752-417c-a8ee-bb847881767a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:02.855813 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.855791    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-kube-api-access-qmh8p" pod "6728c86f-880f-4ea6-8755-7bc9c01d4981" (UID: "6728c86f-880f-4ea6-8755-7bc9c01d4981"). InnerVolumeSpecName "kube-api-access-qmh8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:28:02.943137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943104    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943136    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943151    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943164    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943176    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943188    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmh8p\" (UniqueName: \"kubernetes.io/projected/6728c86f-880f-4ea6-8755-7bc9c01d4981-kube-api-access-qmh8p\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943201    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943213    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txvt\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-kube-api-access-2txvt\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943243    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943257    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6728c86f-880f-4ea6-8755-7bc9c01d4981-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943288    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3909073-3752-417c-a8ee-bb847881767a-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943301    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3909073-3752-417c-a8ee-bb847881767a-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943316    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943329    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3909073-3752-417c-a8ee-bb847881767a-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:02.943613 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:02.943343    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3909073-3752-417c-a8ee-bb847881767a-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:28:03.231914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.231878    4931 generic.go:356] "Generic (PLEG): container finished" podID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerID="67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1" exitCode=0
Nov 01 09:28:03.232123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.231938    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"
Nov 01 09:28:03.232123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.231961    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" event={"ID":"9c47ab33-e553-4f5c-85c8-7c806afc42e2","Type":"ContainerDied","Data":"67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1"}
Nov 01 09:28:03.232123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.232070    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld" event={"ID":"9c47ab33-e553-4f5c-85c8-7c806afc42e2","Type":"ContainerDied","Data":"f54986678e3c8a9a9ce276355f1aab34fb1b6aecd7699ede94287b4553f9fb17"}
Nov 01 09:28:03.232123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.232094    4931 scope.go:122] "RemoveContainer" containerID="67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1"
Nov 01 09:28:03.233660 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.233639    4931 generic.go:356] "Generic (PLEG): container finished" podID="d601123f-a55b-49a6-8271-2b9877866450" containerID="c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83" exitCode=0
Nov 01 09:28:03.233780 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.233708    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"
Nov 01 09:28:03.233780 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.233728    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" event={"ID":"d601123f-a55b-49a6-8271-2b9877866450","Type":"ContainerDied","Data":"c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83"}
Nov 01 09:28:03.233883 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.233779    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf" event={"ID":"d601123f-a55b-49a6-8271-2b9877866450","Type":"ContainerDied","Data":"1d0aa20bf0c231c13cb68601b3f07f5cc202dd13b3773f331d828c5d03de038c"}
Nov 01 09:28:03.238625 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.238595    4931 generic.go:356] "Generic (PLEG): container finished" podID="b3909073-3752-417c-a8ee-bb847881767a" containerID="8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce" exitCode=0
Nov 01 09:28:03.239056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.238658    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"
Nov 01 09:28:03.239056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.238689    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" event={"ID":"b3909073-3752-417c-a8ee-bb847881767a","Type":"ContainerDied","Data":"8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce"}
Nov 01 09:28:03.239056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.238739    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr" event={"ID":"b3909073-3752-417c-a8ee-bb847881767a","Type":"ContainerDied","Data":"8f87a6d1762585ab242c0cb4c3801cedff4769d4bf36c40adfda176934aa0beb"}
Nov 01 09:28:03.244154 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.244110    4931 generic.go:356] "Generic (PLEG): container finished" podID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerID="2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d" exitCode=0
Nov 01 09:28:03.244548 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.244176    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"
Nov 01 09:28:03.244548 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.244177    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" event={"ID":"6728c86f-880f-4ea6-8755-7bc9c01d4981","Type":"ContainerDied","Data":"2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d"}
Nov 01 09:28:03.244548 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.244232    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f" event={"ID":"6728c86f-880f-4ea6-8755-7bc9c01d4981","Type":"ContainerDied","Data":"53ef8d4a80e7b9af6a5910a113dd214b27bec602b419fe7da66f3146ddc74adb"}
Nov 01 09:28:03.245305 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.245287    4931 scope.go:122] "RemoveContainer" containerID="67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1"
Nov 01 09:28:03.245623 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:28:03.245600    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1\": container with ID starting with 67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1 not found: ID does not exist" containerID="67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1"
Nov 01 09:28:03.245685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.245632    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1"} err="failed to get container status \"67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1\": rpc error: code = NotFound desc = could not find container \"67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1\": container with ID starting with 67ac5c6229cff0c310a7716576503d97453ca395961853c9436acb6add9187b1 not found: ID does not exist"
Nov 01 09:28:03.245685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.245648    4931 scope.go:122] "RemoveContainer" containerID="c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83"
Nov 01 09:28:03.256551 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.256533    4931 scope.go:122] "RemoveContainer" containerID="c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83"
Nov 01 09:28:03.256823 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:28:03.256804    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83\": container with ID starting with c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83 not found: ID does not exist" containerID="c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83"
Nov 01 09:28:03.256965 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.256836    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83"} err="failed to get container status \"c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83\": rpc error: code = NotFound desc = could not find container \"c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83\": container with ID starting with c6ba670a0d7c95cb734a025fea66d15a8db7743f835d33b2c275423df4051c83 not found: ID does not exist"
Nov 01 09:28:03.256965 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.256859    4931 scope.go:122] "RemoveContainer" containerID="8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce"
Nov 01 09:28:03.270400 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.266918    4931 scope.go:122] "RemoveContainer" containerID="8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce"
Nov 01 09:28:03.270943 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:28:03.270916    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce\": container with ID starting with 8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce not found: ID does not exist" containerID="8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce"
Nov 01 09:28:03.271120 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.270957    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce"} err="failed to get container status \"8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce\": rpc error: code = NotFound desc = could not find container \"8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce\": container with ID starting with 8f36709792d521785ff990f593328ac3dd2a0caaff403f39211783d9554fcbce not found: ID does not exist"
Nov 01 09:28:03.271120 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.270982    4931 scope.go:122] "RemoveContainer" containerID="2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d"
Nov 01 09:28:03.283727 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.283703    4931 scope.go:122] "RemoveContainer" containerID="2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d"
Nov 01 09:28:03.284070 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:28:03.284019    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d\": container with ID starting with 2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d not found: ID does not exist" containerID="2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d"
Nov 01 09:28:03.284149 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.284085    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d"} err="failed to get container status \"2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d\": rpc error: code = NotFound desc = could not find container \"2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d\": container with ID starting with 2253406de55748cf3c1568c6d7486686f8db7a24638b7b580ca25991b513f87d not found: ID does not exist"
Nov 01 09:28:03.394329 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.394299    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"]
Nov 01 09:28:03.399327 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.399299    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-6vsvf"]
Nov 01 09:28:03.502142 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.502045    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"]
Nov 01 09:28:03.506408 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.506380    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-4nr5f"]
Nov 01 09:28:03.610811 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.610775    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"]
Nov 01 09:28:03.616318 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.616290    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-tl8wr"]
Nov 01 09:28:03.723015 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.722981    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"]
Nov 01 09:28:03.727783 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:03.727755    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-rnwld"]
Nov 01 09:28:04.049312 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:04.049282    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" path="/var/lib/kubelet/pods/6728c86f-880f-4ea6-8755-7bc9c01d4981/volumes"
Nov 01 09:28:04.049776 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:04.049763    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" path="/var/lib/kubelet/pods/9c47ab33-e553-4f5c-85c8-7c806afc42e2/volumes"
Nov 01 09:28:04.050194 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:04.050181    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3909073-3752-417c-a8ee-bb847881767a" path="/var/lib/kubelet/pods/b3909073-3752-417c-a8ee-bb847881767a/volumes"
Nov 01 09:28:04.050909 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:28:04.050895    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d601123f-a55b-49a6-8271-2b9877866450" path="/var/lib/kubelet/pods/d601123f-a55b-49a6-8271-2b9877866450/volumes"
Nov 01 09:28:31.024892 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:28:31.024813    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:29:01.029388 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:29:01.029345    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:29:31.025376 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:29:31.025326    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:29:56.997252 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:29:56.997041    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"]
Nov 01 09:30:00.186363 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.186315    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"]
Nov 01 09:30:00.187108 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187089    4931 cpu_manager.go:414] "containerMap: removing container" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy"
Nov 01 09:30:00.187164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187111    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy"
Nov 01 09:30:00.187164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187127    4931 cpu_manager.go:414] "containerMap: removing container" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy"
Nov 01 09:30:00.187164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187135    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy"
Nov 01 09:30:00.187258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187172    4931 cpu_manager.go:414] "containerMap: removing container" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy"
Nov 01 09:30:00.187258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187180    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy"
Nov 01 09:30:00.187258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187200    4931 cpu_manager.go:414] "containerMap: removing container" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy"
Nov 01 09:30:00.187258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187207    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy"
Nov 01 09:30:00.187258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187222    4931 cpu_manager.go:414] "containerMap: removing container" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy"
Nov 01 09:30:00.187258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187229    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy"
Nov 01 09:30:00.187258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187247    4931 cpu_manager.go:414] "containerMap: removing container" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy"
Nov 01 09:30:00.187258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187253    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy"
Nov 01 09:30:00.187543 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187391    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6728c86f-880f-4ea6-8755-7bc9c01d4981" containerName="istio-proxy"
Nov 01 09:30:00.187543 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187403    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c47ab33-e553-4f5c-85c8-7c806afc42e2" containerName="istio-proxy"
Nov 01 09:30:00.187543 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187415    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy"
Nov 01 09:30:00.187543 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187425    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="91aedd00-ab73-46a1-8ef7-11d6440a4706" containerName="istio-proxy"
Nov 01 09:30:00.187543 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187438    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3909073-3752-417c-a8ee-bb847881767a" containerName="istio-proxy"
Nov 01 09:30:00.187543 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.187450    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d601123f-a55b-49a6-8271-2b9877866450" containerName="istio-proxy"
Nov 01 09:30:00.191346 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.191320    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"]
Nov 01 09:30:00.191543 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.191431    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:00.197059 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.196996    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"]
Nov 01 09:30:00.197230 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.197207    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:00.201254 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.200838    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:00.233931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.233890    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"]
Nov 01 09:30:00.238582 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.238556    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"]
Nov 01 09:30:00.238726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.238679    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:00.242246 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.242220    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:30:00.244548 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.244529    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:00.308080 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.304403    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"]
Nov 01 09:30:00.308080 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.305093    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9xr\" (UniqueName: \"kubernetes.io/projected/9b397369-7fc2-408f-bbfb-5b0823062072-kube-api-access-dn9xr\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs\" (UID: \"9b397369-7fc2-408f-bbfb-5b0823062072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:00.308080 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.305156    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4fdb\" (UniqueName: \"kubernetes.io/projected/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-kube-api-access-k4fdb\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9\" (UID: \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:00.308080 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.305314    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfdf\" (UniqueName: \"kubernetes.io/projected/a75606ff-63bb-4cfc-9c68-1c223b40115e-kube-api-access-smfdf\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g\" (UID: \"a75606ff-63bb-4cfc-9c68-1c223b40115e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:00.308080 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.305374    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b397369-7fc2-408f-bbfb-5b0823062072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs\" (UID: \"9b397369-7fc2-408f-bbfb-5b0823062072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:00.308080 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.305465    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75606ff-63bb-4cfc-9c68-1c223b40115e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g\" (UID: \"a75606ff-63bb-4cfc-9c68-1c223b40115e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:00.308080 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.305625    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9\" (UID: \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:00.309621 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.309590    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"]
Nov 01 09:30:00.314580 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.314553    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"]
Nov 01 09:30:00.352035 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.351987    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"]
Nov 01 09:30:00.356506 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.356473    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"]
Nov 01 09:30:00.406842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.406805    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b397369-7fc2-408f-bbfb-5b0823062072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs\" (UID: \"9b397369-7fc2-408f-bbfb-5b0823062072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:00.407072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.406856    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75606ff-63bb-4cfc-9c68-1c223b40115e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g\" (UID: \"a75606ff-63bb-4cfc-9c68-1c223b40115e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:00.407072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.406925    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2cp\" (UniqueName: \"kubernetes.io/projected/24d5adfd-0c38-4e30-ab0c-f5de89a57121-kube-api-access-cr2cp\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb\" (UID: \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:00.407072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.406951    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892090-hshbw\" (UID: \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:00.407072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.406998    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bprhw\" (UniqueName: \"kubernetes.io/projected/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-kube-api-access-bprhw\") pod \"maas-api-key-cleanup-29892090-hshbw\" (UID: \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:00.407072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.407042    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9\" (UID: \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:00.407354 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.407088    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9xr\" (UniqueName: \"kubernetes.io/projected/9b397369-7fc2-408f-bbfb-5b0823062072-kube-api-access-dn9xr\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs\" (UID: \"9b397369-7fc2-408f-bbfb-5b0823062072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:00.407354 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.407123    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4fdb\" (UniqueName: \"kubernetes.io/projected/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-kube-api-access-k4fdb\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9\" (UID: \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:00.407354 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.407193    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfdf\" (UniqueName: \"kubernetes.io/projected/a75606ff-63bb-4cfc-9c68-1c223b40115e-kube-api-access-smfdf\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g\" (UID: \"a75606ff-63bb-4cfc-9c68-1c223b40115e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:00.407354 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.407209    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d5adfd-0c38-4e30-ab0c-f5de89a57121-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb\" (UID: \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:00.407798 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.407766    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75606ff-63bb-4cfc-9c68-1c223b40115e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g\" (UID: \"a75606ff-63bb-4cfc-9c68-1c223b40115e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:00.407905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.407766    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b397369-7fc2-408f-bbfb-5b0823062072-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs\" (UID: \"9b397369-7fc2-408f-bbfb-5b0823062072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:00.408013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.407986    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9\" (UID: \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:00.468180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.468097    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4fdb\" (UniqueName: \"kubernetes.io/projected/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-kube-api-access-k4fdb\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9\" (UID: \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:00.468180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.468110    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9xr\" (UniqueName: \"kubernetes.io/projected/9b397369-7fc2-408f-bbfb-5b0823062072-kube-api-access-dn9xr\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs\" (UID: \"9b397369-7fc2-408f-bbfb-5b0823062072\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:00.468882 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.468855    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfdf\" (UniqueName: \"kubernetes.io/projected/a75606ff-63bb-4cfc-9c68-1c223b40115e-kube-api-access-smfdf\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g\" (UID: \"a75606ff-63bb-4cfc-9c68-1c223b40115e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:00.508755 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.508724    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bprhw\" (UniqueName: \"kubernetes.io/projected/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-kube-api-access-bprhw\") pod \"maas-api-key-cleanup-29892090-hshbw\" (UID: \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:00.508953 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.508815    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d5adfd-0c38-4e30-ab0c-f5de89a57121-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb\" (UID: \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:00.508953 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.508898    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2cp\" (UniqueName: \"kubernetes.io/projected/24d5adfd-0c38-4e30-ab0c-f5de89a57121-kube-api-access-cr2cp\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb\" (UID: \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:00.508953 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.508902    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:00.508953 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.508932    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892090-hshbw\" (UID: \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:00.509559 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.509536    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d5adfd-0c38-4e30-ab0c-f5de89a57121-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb\" (UID: \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:00.509598 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.509561    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892090-hshbw\" (UID: \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:00.518312 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.518278    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:00.524538 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.524511    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:00.584057 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.580657    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2cp\" (UniqueName: \"kubernetes.io/projected/24d5adfd-0c38-4e30-ab0c-f5de89a57121-kube-api-access-cr2cp\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb\" (UID: \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:00.602969 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.602902    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bprhw\" (UniqueName: \"kubernetes.io/projected/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-kube-api-access-bprhw\") pod \"maas-api-key-cleanup-29892090-hshbw\" (UID: \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:00.856009 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.855977    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:00.861474 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.861441    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:00.907960 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.907930    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"]
Nov 01 09:30:00.911368 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:30:00.911329    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc863aca_8a42_40ce_85d0_d36e8a1d4eb2.slice/crio-f074ef3eb0f2e40bc77b6c960bcb36968253019aeeba2069543f35686e5e3ccf WatchSource:0}: Error finding container f074ef3eb0f2e40bc77b6c960bcb36968253019aeeba2069543f35686e5e3ccf: Status 404 returned error can't find the container with id f074ef3eb0f2e40bc77b6c960bcb36968253019aeeba2069543f35686e5e3ccf
Nov 01 09:30:00.959904 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.958410    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"]
Nov 01 09:30:00.963590 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:00.963557    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"]
Nov 01 09:30:01.052376 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:01.052328    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:30:01.077556 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:01.077516    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" event={"ID":"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2","Type":"ContainerStarted","Data":"f074ef3eb0f2e40bc77b6c960bcb36968253019aeeba2069543f35686e5e3ccf"}
Nov 01 09:30:01.079474 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:01.079437    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs" event={"ID":"9b397369-7fc2-408f-bbfb-5b0823062072","Type":"ContainerStarted","Data":"9ab0da0db64a2026985028552f4ab4115ed1b39b5f6936e55899d31c1a0e85d3"}
Nov 01 09:30:01.081365 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:01.081326    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" event={"ID":"a75606ff-63bb-4cfc-9c68-1c223b40115e","Type":"ContainerStarted","Data":"9c3e92daf24bf5d4da625b16308b9d81d91da87d6a7c5615d0aed382d894eb0d"}
Nov 01 09:30:01.256658 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:01.256579    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"]
Nov 01 09:30:01.257130 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:30:01.256833    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7cf5b1_5eea_43f7_addf_92bd1f2abe93.slice/crio-31f729df8cfa92fe51c1b6816af825d4c147acfc6d594a53a880f9ab4fabc33e WatchSource:0}: Error finding container 31f729df8cfa92fe51c1b6816af825d4c147acfc6d594a53a880f9ab4fabc33e: Status 404 returned error can't find the container with id 31f729df8cfa92fe51c1b6816af825d4c147acfc6d594a53a880f9ab4fabc33e
Nov 01 09:30:01.257781 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:30:01.257758    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d5adfd_0c38_4e30_ab0c_f5de89a57121.slice/crio-f1b693a6c2baa478aa78084c6d24b9502d3c8b6c62b10984f1e2820a3c8b646e WatchSource:0}: Error finding container f1b693a6c2baa478aa78084c6d24b9502d3c8b6c62b10984f1e2820a3c8b646e: Status 404 returned error can't find the container with id f1b693a6c2baa478aa78084c6d24b9502d3c8b6c62b10984f1e2820a3c8b646e
Nov 01 09:30:01.260363 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:01.260336    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"]
Nov 01 09:30:02.092820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.092788    4931 generic.go:356] "Generic (PLEG): container finished" podID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerID="6d8fa8bba9dd50adb966e98debb49193d1c22e46cbd084db349e988c46bbde01" exitCode=60
Nov 01 09:30:02.093010 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.092992    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" event={"ID":"a75606ff-63bb-4cfc-9c68-1c223b40115e","Type":"ContainerDied","Data":"6d8fa8bba9dd50adb966e98debb49193d1c22e46cbd084db349e988c46bbde01"}
Nov 01 09:30:02.093433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.093410    4931 scope.go:122] "RemoveContainer" containerID="6d8fa8bba9dd50adb966e98debb49193d1c22e46cbd084db349e988c46bbde01"
Nov 01 09:30:02.096768 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.096737    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" event={"ID":"24d5adfd-0c38-4e30-ab0c-f5de89a57121","Type":"ContainerStarted","Data":"dc72976292e61a46a017c297e5fe2359e129204116fbcf06f79df4fc7ada1add"}
Nov 01 09:30:02.096895 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.096801    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" event={"ID":"24d5adfd-0c38-4e30-ab0c-f5de89a57121","Type":"ContainerStarted","Data":"f1b693a6c2baa478aa78084c6d24b9502d3c8b6c62b10984f1e2820a3c8b646e"}
Nov 01 09:30:02.099197 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.099168    4931 generic.go:356] "Generic (PLEG): container finished" podID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerID="064cd6ff8d142eb1d4830b36baa4c330cd0851e8ae340e7d5cb1162fd30cd089" exitCode=60
Nov 01 09:30:02.099314 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.099247    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" event={"ID":"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2","Type":"ContainerDied","Data":"064cd6ff8d142eb1d4830b36baa4c330cd0851e8ae340e7d5cb1162fd30cd089"}
Nov 01 09:30:02.099500 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.099480    4931 scope.go:122] "RemoveContainer" containerID="064cd6ff8d142eb1d4830b36baa4c330cd0851e8ae340e7d5cb1162fd30cd089"
Nov 01 09:30:02.101115 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.101090    4931 generic.go:356] "Generic (PLEG): container finished" podID="9b397369-7fc2-408f-bbfb-5b0823062072" containerID="34f88d023bfe4e8a3b3d1510a676f949e763c930e56d37e70ecd550ec8263dc6" exitCode=60
Nov 01 09:30:02.101229 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.101170    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs" event={"ID":"9b397369-7fc2-408f-bbfb-5b0823062072","Type":"ContainerDied","Data":"34f88d023bfe4e8a3b3d1510a676f949e763c930e56d37e70ecd550ec8263dc6"}
Nov 01 09:30:02.101401 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.101385    4931 scope.go:122] "RemoveContainer" containerID="34f88d023bfe4e8a3b3d1510a676f949e763c930e56d37e70ecd550ec8263dc6"
Nov 01 09:30:02.103191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.103166    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" event={"ID":"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93","Type":"ContainerStarted","Data":"0fa21ec3d2c1469031cad7dbdc509d5269b337b56a6689ef020ac349ec878721"}
Nov 01 09:30:02.103278 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.103212    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" event={"ID":"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93","Type":"ContainerStarted","Data":"31f729df8cfa92fe51c1b6816af825d4c147acfc6d594a53a880f9ab4fabc33e"}
Nov 01 09:30:02.190165 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.190135    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"]
Nov 01 09:30:02.554760 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.554711    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" podStartSLOduration=2.554696749 podStartE2EDuration="2.554696749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:02.552408605 +0000 UTC m=+3472.667281574" watchObservedRunningTime="2026-11-01 09:30:02.554696749 +0000 UTC m=+3472.669569716"
Nov 01 09:30:02.777546 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:02.777495    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" podStartSLOduration=2.777475845 podStartE2EDuration="2.777475845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:02.667103594 +0000 UTC m=+3472.781976563" watchObservedRunningTime="2026-11-01 09:30:02.777475845 +0000 UTC m=+3472.892348808"
Nov 01 09:30:03.113357 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.113313    4931 generic.go:356] "Generic (PLEG): container finished" podID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerID="dc72976292e61a46a017c297e5fe2359e129204116fbcf06f79df4fc7ada1add" exitCode=60
Nov 01 09:30:03.113584 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.113443    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" event={"ID":"24d5adfd-0c38-4e30-ab0c-f5de89a57121","Type":"ContainerDied","Data":"dc72976292e61a46a017c297e5fe2359e129204116fbcf06f79df4fc7ada1add"}
Nov 01 09:30:03.113690 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.113671    4931 scope.go:122] "RemoveContainer" containerID="dc72976292e61a46a017c297e5fe2359e129204116fbcf06f79df4fc7ada1add"
Nov 01 09:30:03.117455 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.117319    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" event={"ID":"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2","Type":"ContainerStarted","Data":"131a542960855d40aa5286fb7349b0aed439ab3d7fbfc514a401b1471d1ae4ed"}
Nov 01 09:30:03.122071 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.119799    4931 generic.go:356] "Generic (PLEG): container finished" podID="9b397369-7fc2-408f-bbfb-5b0823062072" containerID="a80fd00eda27cd3b1e53c03a97ce21d0871d80e4126318fe090a20a37e84b4ec" exitCode=60
Nov 01 09:30:03.122071 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.120276    4931 scope.go:122] "RemoveContainer" containerID="a80fd00eda27cd3b1e53c03a97ce21d0871d80e4126318fe090a20a37e84b4ec"
Nov 01 09:30:03.122071 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:03.120539    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs_odh-ai-gateway-infra(9b397369-7fc2-408f-bbfb-5b0823062072)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs" podUID="9b397369-7fc2-408f-bbfb-5b0823062072"
Nov 01 09:30:03.122071 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.120589    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs" event={"ID":"9b397369-7fc2-408f-bbfb-5b0823062072","Type":"ContainerDied","Data":"a80fd00eda27cd3b1e53c03a97ce21d0871d80e4126318fe090a20a37e84b4ec"}
Nov 01 09:30:03.122071 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.120636    4931 scope.go:122] "RemoveContainer" containerID="34f88d023bfe4e8a3b3d1510a676f949e763c930e56d37e70ecd550ec8263dc6"
Nov 01 09:30:03.123513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.123485    4931 generic.go:356] "Generic (PLEG): container finished" podID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerID="0fa21ec3d2c1469031cad7dbdc509d5269b337b56a6689ef020ac349ec878721" exitCode=60
Nov 01 09:30:03.123651 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.123539    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" event={"ID":"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93","Type":"ContainerDied","Data":"0fa21ec3d2c1469031cad7dbdc509d5269b337b56a6689ef020ac349ec878721"}
Nov 01 09:30:03.123880 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.123862    4931 scope.go:122] "RemoveContainer" containerID="0fa21ec3d2c1469031cad7dbdc509d5269b337b56a6689ef020ac349ec878721"
Nov 01 09:30:03.126393 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.126365    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" event={"ID":"a75606ff-63bb-4cfc-9c68-1c223b40115e","Type":"ContainerStarted","Data":"74e3b2e0475bb71190eb11dd2051d9fd90f9494e179f6a7c4456ff6e520fa78b"}
Nov 01 09:30:03.338624 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.338526    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" podStartSLOduration=3.338511592 podStartE2EDuration="3.338511592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:03.336951403 +0000 UTC m=+3473.451824367" watchObservedRunningTime="2026-11-01 09:30:03.338511592 +0000 UTC m=+3473.453384559"
Nov 01 09:30:03.451582 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:03.451526    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" podStartSLOduration=3.451515501 podStartE2EDuration="3.451515501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:03.450431972 +0000 UTC m=+3473.565304937" watchObservedRunningTime="2026-11-01 09:30:03.451515501 +0000 UTC m=+3473.566388488"
Nov 01 09:30:04.136524 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.136491    4931 generic.go:356] "Generic (PLEG): container finished" podID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerID="131a542960855d40aa5286fb7349b0aed439ab3d7fbfc514a401b1471d1ae4ed" exitCode=60
Nov 01 09:30:04.137019 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.136540    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" event={"ID":"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2","Type":"ContainerDied","Data":"131a542960855d40aa5286fb7349b0aed439ab3d7fbfc514a401b1471d1ae4ed"}
Nov 01 09:30:04.137019 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.136603    4931 scope.go:122] "RemoveContainer" containerID="064cd6ff8d142eb1d4830b36baa4c330cd0851e8ae340e7d5cb1162fd30cd089"
Nov 01 09:30:04.137019 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.136900    4931 scope.go:122] "RemoveContainer" containerID="131a542960855d40aa5286fb7349b0aed439ab3d7fbfc514a401b1471d1ae4ed"
Nov 01 09:30:04.137311 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:04.137290    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9_odh-ai-gateway-infra(cc863aca-8a42-40ce-85d0-d36e8a1d4eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2"
Nov 01 09:30:04.138715 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.138686    4931 scope.go:122] "RemoveContainer" containerID="a80fd00eda27cd3b1e53c03a97ce21d0871d80e4126318fe090a20a37e84b4ec"
Nov 01 09:30:04.138996 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:04.138974    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs_odh-ai-gateway-infra(9b397369-7fc2-408f-bbfb-5b0823062072)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs" podUID="9b397369-7fc2-408f-bbfb-5b0823062072"
Nov 01 09:30:04.140303 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.140279    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" event={"ID":"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93","Type":"ContainerStarted","Data":"bf0164dddb1f4a7113c0ed60101fb306349762e5c260d1e122ec377059684216"}
Nov 01 09:30:04.142186 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.142163    4931 generic.go:356] "Generic (PLEG): container finished" podID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerID="74e3b2e0475bb71190eb11dd2051d9fd90f9494e179f6a7c4456ff6e520fa78b" exitCode=60
Nov 01 09:30:04.142306 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.142237    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" event={"ID":"a75606ff-63bb-4cfc-9c68-1c223b40115e","Type":"ContainerDied","Data":"74e3b2e0475bb71190eb11dd2051d9fd90f9494e179f6a7c4456ff6e520fa78b"}
Nov 01 09:30:04.142469 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.142451    4931 scope.go:122] "RemoveContainer" containerID="74e3b2e0475bb71190eb11dd2051d9fd90f9494e179f6a7c4456ff6e520fa78b"
Nov 01 09:30:04.142740 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:04.142721    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g_odh-ai-gateway-infra(a75606ff-63bb-4cfc-9c68-1c223b40115e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e"
Nov 01 09:30:04.144151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.144131    4931 generic.go:356] "Generic (PLEG): container finished" podID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerID="d289980bca49fa8b08f967a4bd60a47058b24282f18f72afb47c636e1617c1b2" exitCode=60
Nov 01 09:30:04.144248 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.144177    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" event={"ID":"24d5adfd-0c38-4e30-ab0c-f5de89a57121","Type":"ContainerDied","Data":"d289980bca49fa8b08f967a4bd60a47058b24282f18f72afb47c636e1617c1b2"}
Nov 01 09:30:04.144441 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.144427    4931 scope.go:122] "RemoveContainer" containerID="d289980bca49fa8b08f967a4bd60a47058b24282f18f72afb47c636e1617c1b2"
Nov 01 09:30:04.144659 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:04.144644    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb_odh-ai-gateway-infra(24d5adfd-0c38-4e30-ab0c-f5de89a57121)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121"
Nov 01 09:30:04.150649 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.150627    4931 scope.go:122] "RemoveContainer" containerID="6d8fa8bba9dd50adb966e98debb49193d1c22e46cbd084db349e988c46bbde01"
Nov 01 09:30:04.166956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:04.166929    4931 scope.go:122] "RemoveContainer" containerID="dc72976292e61a46a017c297e5fe2359e129204116fbcf06f79df4fc7ada1add"
Nov 01 09:30:05.152944 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:05.152908    4931 scope.go:122] "RemoveContainer" containerID="d289980bca49fa8b08f967a4bd60a47058b24282f18f72afb47c636e1617c1b2"
Nov 01 09:30:05.153467 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:05.153322    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb_odh-ai-gateway-infra(24d5adfd-0c38-4e30-ab0c-f5de89a57121)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121"
Nov 01 09:30:05.154628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:05.154606    4931 scope.go:122] "RemoveContainer" containerID="131a542960855d40aa5286fb7349b0aed439ab3d7fbfc514a401b1471d1ae4ed"
Nov 01 09:30:05.154892 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:05.154871    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9_odh-ai-gateway-infra(cc863aca-8a42-40ce-85d0-d36e8a1d4eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2"
Nov 01 09:30:05.156205 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:05.156181    4931 generic.go:356] "Generic (PLEG): container finished" podID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerID="bf0164dddb1f4a7113c0ed60101fb306349762e5c260d1e122ec377059684216" exitCode=60
Nov 01 09:30:05.156313 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:05.156242    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" event={"ID":"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93","Type":"ContainerDied","Data":"bf0164dddb1f4a7113c0ed60101fb306349762e5c260d1e122ec377059684216"}
Nov 01 09:30:05.156313 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:05.156289    4931 scope.go:122] "RemoveContainer" containerID="0fa21ec3d2c1469031cad7dbdc509d5269b337b56a6689ef020ac349ec878721"
Nov 01 09:30:05.156476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:05.156461    4931 scope.go:122] "RemoveContainer" containerID="bf0164dddb1f4a7113c0ed60101fb306349762e5c260d1e122ec377059684216"
Nov 01 09:30:05.156760 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:05.156736    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892090-hshbw_odh-ai-gateway-infra(6c7cf5b1-5eea-43f7-addf-92bd1f2abe93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93"
Nov 01 09:30:05.158823 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:05.158808    4931 scope.go:122] "RemoveContainer" containerID="74e3b2e0475bb71190eb11dd2051d9fd90f9494e179f6a7c4456ff6e520fa78b"
Nov 01 09:30:05.159018 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:05.159002    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g_odh-ai-gateway-infra(a75606ff-63bb-4cfc-9c68-1c223b40115e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e"
Nov 01 09:30:06.169806 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:06.169775    4931 scope.go:122] "RemoveContainer" containerID="bf0164dddb1f4a7113c0ed60101fb306349762e5c260d1e122ec377059684216"
Nov 01 09:30:06.170321 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:06.170177    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892090-hshbw_odh-ai-gateway-infra(6c7cf5b1-5eea-43f7-addf-92bd1f2abe93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93"
Nov 01 09:30:12.023520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.023450    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj" podUID="8958de78-bacf-497f-8051-f949459e57a9" containerName="main" containerID="cri-o://cb17de9c562c7ad2400c4d0a3530dee8a4f4153dd8a3b11985ac63d89786d5d9" gracePeriod=45
Nov 01 09:30:12.120806 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.120754    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj" podUID="8958de78-bacf-497f-8051-f949459e57a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": dial tcp 10.133.0.108:8000: connect: connection refused"
Nov 01 09:30:12.229988 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.229912    4931 generic.go:356] "Generic (PLEG): container finished" podID="8958de78-bacf-497f-8051-f949459e57a9" containerID="cb17de9c562c7ad2400c4d0a3530dee8a4f4153dd8a3b11985ac63d89786d5d9" exitCode=0
Nov 01 09:30:12.229988 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.229990    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj" event={"ID":"8958de78-bacf-497f-8051-f949459e57a9","Type":"ContainerDied","Data":"cb17de9c562c7ad2400c4d0a3530dee8a4f4153dd8a3b11985ac63d89786d5d9"}
Nov 01 09:30:12.310726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.310698    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:30:12.363457 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363421    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-dshm\") pod \"8958de78-bacf-497f-8051-f949459e57a9\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") "
Nov 01 09:30:12.363659 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363490    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-home\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-home\") pod \"8958de78-bacf-497f-8051-f949459e57a9\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") "
Nov 01 09:30:12.363659 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363578    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8958de78-bacf-497f-8051-f949459e57a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8958de78-bacf-497f-8051-f949459e57a9-tls-certs\") pod \"8958de78-bacf-497f-8051-f949459e57a9\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") "
Nov 01 09:30:12.363659 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363609    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-tmp-dir\") pod \"8958de78-bacf-497f-8051-f949459e57a9\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") "
Nov 01 09:30:12.363820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363663    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-model-cache\") pod \"8958de78-bacf-497f-8051-f949459e57a9\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") "
Nov 01 09:30:12.363820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363705    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8958de78-bacf-497f-8051-f949459e57a9-kube-api-access-sh7pw\" (UniqueName: \"kubernetes.io/projected/8958de78-bacf-497f-8051-f949459e57a9-kube-api-access-sh7pw\") pod \"8958de78-bacf-497f-8051-f949459e57a9\" (UID: \"8958de78-bacf-497f-8051-f949459e57a9\") "
Nov 01 09:30:12.363820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363784    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-home" pod "8958de78-bacf-497f-8051-f949459e57a9" (UID: "8958de78-bacf-497f-8051-f949459e57a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:12.363978 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363880    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-tmp-dir" pod "8958de78-bacf-497f-8051-f949459e57a9" (UID: "8958de78-bacf-497f-8051-f949459e57a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:12.363978 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.363890    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-model-cache" pod "8958de78-bacf-497f-8051-f949459e57a9" (UID: "8958de78-bacf-497f-8051-f949459e57a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:12.364315 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.364295    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:12.364315 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.364315    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:12.364634 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.364329    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:12.366260 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.366230    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8958de78-bacf-497f-8051-f949459e57a9-kube-api-access-sh7pw" pod "8958de78-bacf-497f-8051-f949459e57a9" (UID: "8958de78-bacf-497f-8051-f949459e57a9"). InnerVolumeSpecName "kube-api-access-sh7pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:12.366392 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.366375    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8958de78-bacf-497f-8051-f949459e57a9-tls-certs" pod "8958de78-bacf-497f-8051-f949459e57a9" (UID: "8958de78-bacf-497f-8051-f949459e57a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:30:12.366441 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.366388    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-dshm" pod "8958de78-bacf-497f-8051-f949459e57a9" (UID: "8958de78-bacf-497f-8051-f949459e57a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:12.466482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.466436    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8958de78-bacf-497f-8051-f949459e57a9-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:12.466482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.466479    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8958de78-bacf-497f-8051-f949459e57a9-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:12.466754 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.466497    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh7pw\" (UniqueName: \"kubernetes.io/projected/8958de78-bacf-497f-8051-f949459e57a9-kube-api-access-sh7pw\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:12.615470 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.615427    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7415-8eae4b"]
Nov 01 09:30:12.616080 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.616065    4931 cpu_manager.go:414] "containerMap: removing container" podUID="8958de78-bacf-497f-8051-f949459e57a9" containerName="main"
Nov 01 09:30:12.616128 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.616082    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8958de78-bacf-497f-8051-f949459e57a9" containerName="main"
Nov 01 09:30:12.616212 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.616203    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8958de78-bacf-497f-8051-f949459e57a9" containerName="main"
Nov 01 09:30:12.619790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.619772    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:12.670049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.669104    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwz6j\" (UniqueName: \"kubernetes.io/projected/7a0a310c-67dd-4274-abb5-c08ce59f49a1-kube-api-access-jwz6j\") pod \"test-curl-7415-8eae4b\" (UID: \"7a0a310c-67dd-4274-abb5-c08ce59f49a1\") " pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:12.670049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.669893    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-8eae4b"]
Nov 01 09:30:12.670255 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.670081    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-44wvn\""
Nov 01 09:30:12.770383 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.770347    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwz6j\" (UniqueName: \"kubernetes.io/projected/7a0a310c-67dd-4274-abb5-c08ce59f49a1-kube-api-access-jwz6j\") pod \"test-curl-7415-8eae4b\" (UID: \"7a0a310c-67dd-4274-abb5-c08ce59f49a1\") " pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:12.829157 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.829120    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwz6j\" (UniqueName: \"kubernetes.io/projected/7a0a310c-67dd-4274-abb5-c08ce59f49a1-kube-api-access-jwz6j\") pod \"test-curl-7415-8eae4b\" (UID: \"7a0a310c-67dd-4274-abb5-c08ce59f49a1\") " pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:12.933211 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:12.933117    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:13.248335 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:13.248242    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj" event={"ID":"8958de78-bacf-497f-8051-f949459e57a9","Type":"ContainerDied","Data":"accc624004265c1e49cf552118d2958f818836d1154ac2436121299f22fadc29"}
Nov 01 09:30:13.248335 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:13.248298    4931 scope.go:122] "RemoveContainer" containerID="cb17de9c562c7ad2400c4d0a3530dee8a4f4153dd8a3b11985ac63d89786d5d9"
Nov 01 09:30:13.248864 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:13.248495    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"
Nov 01 09:30:13.307094 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:13.306696    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-8eae4b"]
Nov 01 09:30:13.308481 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:30:13.308447    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a0a310c_67dd_4274_abb5_c08ce59f49a1.slice/crio-2c9de895990812936418311fad99d347e4b2a1a2658aba7b0bc8166aa97c4ca3 WatchSource:0}: Error finding container 2c9de895990812936418311fad99d347e4b2a1a2658aba7b0bc8166aa97c4ca3: Status 404 returned error can't find the container with id 2c9de895990812936418311fad99d347e4b2a1a2658aba7b0bc8166aa97c4ca3
Nov 01 09:30:13.310809 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:13.310787    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:30:13.408194 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:13.408158    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"]
Nov 01 09:30:13.411685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:13.411659    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-035625/auth-test-model-035625-kserve-6f8f58c95c-f4nnj"]
Nov 01 09:30:14.049708 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:14.049666    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8958de78-bacf-497f-8051-f949459e57a9" path="/var/lib/kubelet/pods/8958de78-bacf-497f-8051-f949459e57a9/volumes"
Nov 01 09:30:14.257485 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:14.257447    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-8eae4b" event={"ID":"7a0a310c-67dd-4274-abb5-c08ce59f49a1","Type":"ContainerStarted","Data":"2c9de895990812936418311fad99d347e4b2a1a2658aba7b0bc8166aa97c4ca3"}
Nov 01 09:30:15.041060 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:15.040987    4931 scope.go:122] "RemoveContainer" containerID="131a542960855d40aa5286fb7349b0aed439ab3d7fbfc514a401b1471d1ae4ed"
Nov 01 09:30:15.041245 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:15.041170    4931 scope.go:122] "RemoveContainer" containerID="d289980bca49fa8b08f967a4bd60a47058b24282f18f72afb47c636e1617c1b2"
Nov 01 09:30:15.041808 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:15.041463    4931 scope.go:122] "RemoveContainer" containerID="bf0164dddb1f4a7113c0ed60101fb306349762e5c260d1e122ec377059684216"
Nov 01 09:30:16.041570 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.041538    4931 scope.go:122] "RemoveContainer" containerID="74e3b2e0475bb71190eb11dd2051d9fd90f9494e179f6a7c4456ff6e520fa78b"
Nov 01 09:30:16.279379 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.279341    4931 generic.go:356] "Generic (PLEG): container finished" podID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerID="85751655a3e84ed38f4befb381bed1929b1a0dad721673c5ae1822c699bc429c" exitCode=60
Nov 01 09:30:16.279561 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.279410    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" event={"ID":"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93","Type":"ContainerDied","Data":"85751655a3e84ed38f4befb381bed1929b1a0dad721673c5ae1822c699bc429c"}
Nov 01 09:30:16.279561 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.279474    4931 scope.go:122] "RemoveContainer" containerID="bf0164dddb1f4a7113c0ed60101fb306349762e5c260d1e122ec377059684216"
Nov 01 09:30:16.279990 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.279962    4931 scope.go:122] "RemoveContainer" containerID="85751655a3e84ed38f4befb381bed1929b1a0dad721673c5ae1822c699bc429c"
Nov 01 09:30:16.280373 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:16.280349    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892090-hshbw_odh-ai-gateway-infra(6c7cf5b1-5eea-43f7-addf-92bd1f2abe93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93"
Nov 01 09:30:16.282697 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.282666    4931 generic.go:356] "Generic (PLEG): container finished" podID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerID="6bc4098c29766c4042c64516a65dec3d6cbab04fa3690cc649b253ba2d78ad89" exitCode=60
Nov 01 09:30:16.282820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.282737    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" event={"ID":"24d5adfd-0c38-4e30-ab0c-f5de89a57121","Type":"ContainerDied","Data":"6bc4098c29766c4042c64516a65dec3d6cbab04fa3690cc649b253ba2d78ad89"}
Nov 01 09:30:16.283286 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.283265    4931 scope.go:122] "RemoveContainer" containerID="6bc4098c29766c4042c64516a65dec3d6cbab04fa3690cc649b253ba2d78ad89"
Nov 01 09:30:16.283639 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:16.283613    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb_odh-ai-gateway-infra(24d5adfd-0c38-4e30-ab0c-f5de89a57121)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121"
Nov 01 09:30:16.285100 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.285065    4931 generic.go:356] "Generic (PLEG): container finished" podID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerID="6b7b26698e92f0cbfe97956bccd500373315f5e50368e310822ed4859b147cab" exitCode=60
Nov 01 09:30:16.285179 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.285108    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" event={"ID":"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2","Type":"ContainerDied","Data":"6b7b26698e92f0cbfe97956bccd500373315f5e50368e310822ed4859b147cab"}
Nov 01 09:30:16.285566 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.285550    4931 scope.go:122] "RemoveContainer" containerID="6b7b26698e92f0cbfe97956bccd500373315f5e50368e310822ed4859b147cab"
Nov 01 09:30:16.285894 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:16.285874    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9_odh-ai-gateway-infra(cc863aca-8a42-40ce-85d0-d36e8a1d4eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2"
Nov 01 09:30:16.426327 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.426300    4931 scope.go:122] "RemoveContainer" containerID="d289980bca49fa8b08f967a4bd60a47058b24282f18f72afb47c636e1617c1b2"
Nov 01 09:30:16.445191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:16.445163    4931 scope.go:122] "RemoveContainer" containerID="131a542960855d40aa5286fb7349b0aed439ab3d7fbfc514a401b1471d1ae4ed"
Nov 01 09:30:17.041787 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.041700    4931 scope.go:122] "RemoveContainer" containerID="a80fd00eda27cd3b1e53c03a97ce21d0871d80e4126318fe090a20a37e84b4ec"
Nov 01 09:30:17.218929 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.218879    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k" podUID="824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" containerName="main" containerID="cri-o://19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138" gracePeriod=45
Nov 01 09:30:17.298710 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.298633    4931 generic.go:356] "Generic (PLEG): container finished" podID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerID="071bdab8779ef4173e876055a60c3e0c0e1a0174084d36172429978e4d657002" exitCode=60
Nov 01 09:30:17.298884 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.298701    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" event={"ID":"a75606ff-63bb-4cfc-9c68-1c223b40115e","Type":"ContainerDied","Data":"071bdab8779ef4173e876055a60c3e0c0e1a0174084d36172429978e4d657002"}
Nov 01 09:30:17.298884 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.298760    4931 scope.go:122] "RemoveContainer" containerID="74e3b2e0475bb71190eb11dd2051d9fd90f9494e179f6a7c4456ff6e520fa78b"
Nov 01 09:30:17.299203 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.299171    4931 scope.go:122] "RemoveContainer" containerID="071bdab8779ef4173e876055a60c3e0c0e1a0174084d36172429978e4d657002"
Nov 01 09:30:17.299528 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:17.299503    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g_odh-ai-gateway-infra(a75606ff-63bb-4cfc-9c68-1c223b40115e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e"
Nov 01 09:30:17.300466 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.300440    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-8eae4b" event={"ID":"7a0a310c-67dd-4274-abb5-c08ce59f49a1","Type":"ContainerStarted","Data":"7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6"}
Nov 01 09:30:17.409014 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.408974    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"]
Nov 01 09:30:17.507852 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.507824    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:30:17.512214 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.512181    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"]
Nov 01 09:30:17.603918 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.603888    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:17.622371 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.622329    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"]
Nov 01 09:30:17.638297 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.638239    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7415-8eae4b" podStartSLOduration=2.570512946 podStartE2EDuration="5.638222023s" totalImagesPullingTime="3.067709077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:17.636946416 +0000 UTC m=+3487.751819387" watchObservedRunningTime="2026-11-01 09:30:17.638222023 +0000 UTC m=+3487.753094992"
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.645577    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-kube-api-access-xpvxx\" (UniqueName: \"kubernetes.io/projected/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-kube-api-access-xpvxx\") pod \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") "
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.645627    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-dshm\") pod \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") "
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.645740    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-model-cache\") pod \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") "
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.645821    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tls-certs\") pod \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") "
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.645861    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tmp-dir\") pod \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") "
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.645901    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-kube-api-access-bprhw\" (UniqueName: \"kubernetes.io/projected/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-kube-api-access-bprhw\") pod \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\" (UID: \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\") "
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.645936    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-home\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-home\") pod \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\" (UID: \"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0\") "
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.645984    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-maas-api-service-ca\") pod \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\" (UID: \"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93\") "
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.646760    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-maas-api-service-ca" pod "6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" (UID: "6c7cf5b1-5eea-43f7-addf-92bd1f2abe93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:17.647158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.646941    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tmp-dir" pod "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" (UID: "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:17.649929 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.649278    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-home" pod "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" (UID: "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:17.649929 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.649649    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-model-cache" pod "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" (UID: "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:17.650278 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.650237    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tls-certs" pod "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" (UID: "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:30:17.651467 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.651433    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-dshm" pod "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" (UID: "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:17.651657 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.651635    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-kube-api-access-bprhw" pod "6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" (UID: "6c7cf5b1-5eea-43f7-addf-92bd1f2abe93"). InnerVolumeSpecName "kube-api-access-bprhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:17.652191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.652175    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-kube-api-access-xpvxx" pod "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" (UID: "824c6bdd-5ccc-4e8b-89b7-b23d901b36f0"). InnerVolumeSpecName "kube-api-access-xpvxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:17.709778 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.709745    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:17.747869 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.747831    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpvxx\" (UniqueName: \"kubernetes.io/projected/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-kube-api-access-xpvxx\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.747869 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.747868    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.748153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.747881    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.748153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.747895    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.748153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.747907    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.748153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.747918    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bprhw\" (UniqueName: \"kubernetes.io/projected/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-kube-api-access-bprhw\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.748153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.747932    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.748153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.747946    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.803469 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.803427    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:17.849383 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.849291    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d5adfd-0c38-4e30-ab0c-f5de89a57121-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d5adfd-0c38-4e30-ab0c-f5de89a57121-maas-api-service-ca\") pod \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\" (UID: \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\") "
Nov 01 09:30:17.849383 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.849347    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-kube-api-access-k4fdb\" (UniqueName: \"kubernetes.io/projected/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-kube-api-access-k4fdb\") pod \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\" (UID: \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\") "
Nov 01 09:30:17.849594 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.849414    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d5adfd-0c38-4e30-ab0c-f5de89a57121-kube-api-access-cr2cp\" (UniqueName: \"kubernetes.io/projected/24d5adfd-0c38-4e30-ab0c-f5de89a57121-kube-api-access-cr2cp\") pod \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\" (UID: \"24d5adfd-0c38-4e30-ab0c-f5de89a57121\") "
Nov 01 09:30:17.849594 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.849509    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-maas-api-service-ca\") pod \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\" (UID: \"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2\") "
Nov 01 09:30:17.849796 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.849774    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d5adfd-0c38-4e30-ab0c-f5de89a57121-maas-api-service-ca" pod "24d5adfd-0c38-4e30-ab0c-f5de89a57121" (UID: "24d5adfd-0c38-4e30-ab0c-f5de89a57121"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:17.849871 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.849849    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-maas-api-service-ca" pod "cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" (UID: "cc863aca-8a42-40ce-85d0-d36e8a1d4eb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:17.850002 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.849986    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.850274 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.850008    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d5adfd-0c38-4e30-ab0c-f5de89a57121-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.851756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.851727    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-kube-api-access-k4fdb" pod "cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" (UID: "cc863aca-8a42-40ce-85d0-d36e8a1d4eb2"). InnerVolumeSpecName "kube-api-access-k4fdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:17.851852 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.851761    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d5adfd-0c38-4e30-ab0c-f5de89a57121-kube-api-access-cr2cp" pod "24d5adfd-0c38-4e30-ab0c-f5de89a57121" (UID: "24d5adfd-0c38-4e30-ab0c-f5de89a57121"). InnerVolumeSpecName "kube-api-access-cr2cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:17.951423 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.951377    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr2cp\" (UniqueName: \"kubernetes.io/projected/24d5adfd-0c38-4e30-ab0c-f5de89a57121-kube-api-access-cr2cp\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:17.951423 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:17.951410    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4fdb\" (UniqueName: \"kubernetes.io/projected/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2-kube-api-access-k4fdb\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:18.310970 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.310941    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"
Nov 01 09:30:18.311480 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.311102    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9" event={"ID":"cc863aca-8a42-40ce-85d0-d36e8a1d4eb2","Type":"ContainerDied","Data":"f074ef3eb0f2e40bc77b6c960bcb36968253019aeeba2069543f35686e5e3ccf"}
Nov 01 09:30:18.311480 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.311173    4931 scope.go:122] "RemoveContainer" containerID="6b7b26698e92f0cbfe97956bccd500373315f5e50368e310822ed4859b147cab"
Nov 01 09:30:18.314008 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.313982    4931 generic.go:356] "Generic (PLEG): container finished" podID="9b397369-7fc2-408f-bbfb-5b0823062072" containerID="6fdc7feac5e0efb1d3142eb106f456e7366825fe305ad65784ed028e10abc439" exitCode=60
Nov 01 09:30:18.314303 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.314279    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs" event={"ID":"9b397369-7fc2-408f-bbfb-5b0823062072","Type":"ContainerDied","Data":"6fdc7feac5e0efb1d3142eb106f456e7366825fe305ad65784ed028e10abc439"}
Nov 01 09:30:18.314775 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.314746    4931 scope.go:122] "RemoveContainer" containerID="6fdc7feac5e0efb1d3142eb106f456e7366825fe305ad65784ed028e10abc439"
Nov 01 09:30:18.315082 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:18.315061    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs_odh-ai-gateway-infra(9b397369-7fc2-408f-bbfb-5b0823062072)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs" podUID="9b397369-7fc2-408f-bbfb-5b0823062072"
Nov 01 09:30:18.316686 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.316663    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw" event={"ID":"6c7cf5b1-5eea-43f7-addf-92bd1f2abe93","Type":"ContainerDied","Data":"31f729df8cfa92fe51c1b6816af825d4c147acfc6d594a53a880f9ab4fabc33e"}
Nov 01 09:30:18.316798 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.316752    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"
Nov 01 09:30:18.338076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.320859    4931 generic.go:356] "Generic (PLEG): container finished" podID="824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" containerID="19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138" exitCode=0
Nov 01 09:30:18.338076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.328067    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"
Nov 01 09:30:18.338076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.328140    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k" event={"ID":"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0","Type":"ContainerDied","Data":"19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138"}
Nov 01 09:30:18.338076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.328184    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k" event={"ID":"824c6bdd-5ccc-4e8b-89b7-b23d901b36f0","Type":"ContainerDied","Data":"0789c73e3e1b2537fa1bda49ea2523fdaabb6cf0384af29f4502d2c9477fecc1"}
Nov 01 09:30:18.338076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.328882    4931 scope.go:122] "RemoveContainer" containerID="a80fd00eda27cd3b1e53c03a97ce21d0871d80e4126318fe090a20a37e84b4ec"
Nov 01 09:30:18.338076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.330747    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"
Nov 01 09:30:18.338076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.330791    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb" event={"ID":"24d5adfd-0c38-4e30-ab0c-f5de89a57121","Type":"ContainerDied","Data":"f1b693a6c2baa478aa78084c6d24b9502d3c8b6c62b10984f1e2820a3c8b646e"}
Nov 01 09:30:18.344611 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.344590    4931 scope.go:122] "RemoveContainer" containerID="85751655a3e84ed38f4befb381bed1929b1a0dad721673c5ae1822c699bc429c"
Nov 01 09:30:18.354479 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.354455    4931 scope.go:122] "RemoveContainer" containerID="19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138"
Nov 01 09:30:18.364612 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.364585    4931 scope.go:122] "RemoveContainer" containerID="19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138"
Nov 01 09:30:18.364909 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:18.364886    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138\": container with ID starting with 19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138 not found: ID does not exist" containerID="19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138"
Nov 01 09:30:18.365056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.364928    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138"} err="failed to get container status \"19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138\": rpc error: code = NotFound desc = could not find container \"19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138\": container with ID starting with 19852c751677be29dbabffeb31f8b3e2fffd7a18daf3ecf27ced6bff82def138 not found: ID does not exist"
Nov 01 09:30:18.365056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.364950    4931 scope.go:122] "RemoveContainer" containerID="6bc4098c29766c4042c64516a65dec3d6cbab04fa3690cc649b253ba2d78ad89"
Nov 01 09:30:18.424293 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.424208    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"]
Nov 01 09:30:18.479403 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.479374    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"]
Nov 01 09:30:18.487796 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.487422    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5b96ac/auth-test-model-5b96ac-kserve-6d5c5776f7-c8v7k"]
Nov 01 09:30:18.586403 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.586373    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:18.658168 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.658129    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75606ff-63bb-4cfc-9c68-1c223b40115e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75606ff-63bb-4cfc-9c68-1c223b40115e-maas-api-service-ca\") pod \"a75606ff-63bb-4cfc-9c68-1c223b40115e\" (UID: \"a75606ff-63bb-4cfc-9c68-1c223b40115e\") "
Nov 01 09:30:18.658380 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.658288    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75606ff-63bb-4cfc-9c68-1c223b40115e-kube-api-access-smfdf\" (UniqueName: \"kubernetes.io/projected/a75606ff-63bb-4cfc-9c68-1c223b40115e-kube-api-access-smfdf\") pod \"a75606ff-63bb-4cfc-9c68-1c223b40115e\" (UID: \"a75606ff-63bb-4cfc-9c68-1c223b40115e\") "
Nov 01 09:30:18.658564 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.658539    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75606ff-63bb-4cfc-9c68-1c223b40115e-maas-api-service-ca" pod "a75606ff-63bb-4cfc-9c68-1c223b40115e" (UID: "a75606ff-63bb-4cfc-9c68-1c223b40115e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:18.658797 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.658781    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75606ff-63bb-4cfc-9c68-1c223b40115e-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:18.660716 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.660682    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75606ff-63bb-4cfc-9c68-1c223b40115e-kube-api-access-smfdf" pod "a75606ff-63bb-4cfc-9c68-1c223b40115e" (UID: "a75606ff-63bb-4cfc-9c68-1c223b40115e"). InnerVolumeSpecName "kube-api-access-smfdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:18.703099 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.702985    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"]
Nov 01 09:30:18.707878 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.707845    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892090-2b8zb"]
Nov 01 09:30:18.760223 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.760186    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smfdf\" (UniqueName: \"kubernetes.io/projected/a75606ff-63bb-4cfc-9c68-1c223b40115e-kube-api-access-smfdf\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:18.810533 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.810499    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"]
Nov 01 09:30:18.813443 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.813418    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892090-hshbw"]
Nov 01 09:30:18.921344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.918616    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"]
Nov 01 09:30:18.933307 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:18.933255    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892090-dgnr9"]
Nov 01 09:30:19.142336 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.142297    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7415-8eae4b"]
Nov 01 09:30:19.146535 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.146506    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7415-8eae4b"]
Nov 01 09:30:19.339880 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.339845    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"
Nov 01 09:30:19.339880 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.339855    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g" event={"ID":"a75606ff-63bb-4cfc-9c68-1c223b40115e","Type":"ContainerDied","Data":"9c3e92daf24bf5d4da625b16308b9d81d91da87d6a7c5615d0aed382d894eb0d"}
Nov 01 09:30:19.340409 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.339928    4931 scope.go:122] "RemoveContainer" containerID="071bdab8779ef4173e876055a60c3e0c0e1a0174084d36172429978e4d657002"
Nov 01 09:30:19.346016 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.345986    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7415-8eae4b" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" containerName="test-curl-7415-8eae4b" containerID="cri-o://7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6" gracePeriod=2
Nov 01 09:30:19.439484 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:19.439390    4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-8eae4b\" is forbidden: User \"system:node:ip-10-0-143-244.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.us-west-2.compute.internal' and this object" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:19.547401 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.547369    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"]
Nov 01 09:30:19.556320 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.556288    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892090-gqj4g"]
Nov 01 09:30:19.604807 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.604767    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"]
Nov 01 09:30:19.768902 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.768869    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:19.769465 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:19.769438    4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-8eae4b\" is forbidden: User \"system:node:ip-10-0-143-244.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.us-west-2.compute.internal' and this object" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:19.818392 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:19.818355    4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-8eae4b\" is forbidden: User \"system:node:ip-10-0-143-244.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.us-west-2.compute.internal' and this object" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:19.875677 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.875634    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b397369-7fc2-408f-bbfb-5b0823062072-kube-api-access-dn9xr\" (UniqueName: \"kubernetes.io/projected/9b397369-7fc2-408f-bbfb-5b0823062072-kube-api-access-dn9xr\") pod \"9b397369-7fc2-408f-bbfb-5b0823062072\" (UID: \"9b397369-7fc2-408f-bbfb-5b0823062072\") "
Nov 01 09:30:19.875852 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.875764    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b397369-7fc2-408f-bbfb-5b0823062072-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b397369-7fc2-408f-bbfb-5b0823062072-maas-api-service-ca\") pod \"9b397369-7fc2-408f-bbfb-5b0823062072\" (UID: \"9b397369-7fc2-408f-bbfb-5b0823062072\") "
Nov 01 09:30:19.876981 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.876950    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b397369-7fc2-408f-bbfb-5b0823062072-maas-api-service-ca" pod "9b397369-7fc2-408f-bbfb-5b0823062072" (UID: "9b397369-7fc2-408f-bbfb-5b0823062072"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:30:19.879469 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.879439    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b397369-7fc2-408f-bbfb-5b0823062072-kube-api-access-dn9xr" pod "9b397369-7fc2-408f-bbfb-5b0823062072" (UID: "9b397369-7fc2-408f-bbfb-5b0823062072"). InnerVolumeSpecName "kube-api-access-dn9xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:19.976912 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.976881    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn9xr\" (UniqueName: \"kubernetes.io/projected/9b397369-7fc2-408f-bbfb-5b0823062072-kube-api-access-dn9xr\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:19.976912 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:19.976910    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b397369-7fc2-408f-bbfb-5b0823062072-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:20.049795 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.049719    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" path="/var/lib/kubelet/pods/24d5adfd-0c38-4e30-ab0c-f5de89a57121/volumes"
Nov 01 09:30:20.050175 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.050161    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" path="/var/lib/kubelet/pods/6c7cf5b1-5eea-43f7-addf-92bd1f2abe93/volumes"
Nov 01 09:30:20.050567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.050556    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" path="/var/lib/kubelet/pods/824c6bdd-5ccc-4e8b-89b7-b23d901b36f0/volumes"
Nov 01 09:30:20.051275 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.051258    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" path="/var/lib/kubelet/pods/a75606ff-63bb-4cfc-9c68-1c223b40115e/volumes"
Nov 01 09:30:20.051671 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.051658    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" path="/var/lib/kubelet/pods/cc863aca-8a42-40ce-85d0-d36e8a1d4eb2/volumes"
Nov 01 09:30:20.093451 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:20.093402    4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-8eae4b\" is forbidden: User \"system:node:ip-10-0-143-244.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.us-west-2.compute.internal' and this object" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:20.354571 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.354538    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"
Nov 01 09:30:20.354571 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.354558    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs" event={"ID":"9b397369-7fc2-408f-bbfb-5b0823062072","Type":"ContainerDied","Data":"9ab0da0db64a2026985028552f4ab4115ed1b39b5f6936e55899d31c1a0e85d3"}
Nov 01 09:30:20.355150 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.354620    4931 scope.go:122] "RemoveContainer" containerID="6fdc7feac5e0efb1d3142eb106f456e7366825fe305ad65784ed028e10abc439"
Nov 01 09:30:20.513801 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.513765    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"]
Nov 01 09:30:20.519397 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:20.519368    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892090-pshcs"]
Nov 01 09:30:21.679122 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:21.679096    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:21.815695 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:21.815657    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0a310c-67dd-4274-abb5-c08ce59f49a1-kube-api-access-jwz6j\" (UniqueName: \"kubernetes.io/projected/7a0a310c-67dd-4274-abb5-c08ce59f49a1-kube-api-access-jwz6j\") pod \"7a0a310c-67dd-4274-abb5-c08ce59f49a1\" (UID: \"7a0a310c-67dd-4274-abb5-c08ce59f49a1\") "
Nov 01 09:30:21.818096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:21.818049    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0a310c-67dd-4274-abb5-c08ce59f49a1-kube-api-access-jwz6j" pod "7a0a310c-67dd-4274-abb5-c08ce59f49a1" (UID: "7a0a310c-67dd-4274-abb5-c08ce59f49a1"). InnerVolumeSpecName "kube-api-access-jwz6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:21.917307 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:21.917213    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwz6j\" (UniqueName: \"kubernetes.io/projected/7a0a310c-67dd-4274-abb5-c08ce59f49a1-kube-api-access-jwz6j\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:22.049410 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:22.049364    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" path="/var/lib/kubelet/pods/7a0a310c-67dd-4274-abb5-c08ce59f49a1/volumes"
Nov 01 09:30:22.049867 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:22.049848    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" path="/var/lib/kubelet/pods/9b397369-7fc2-408f-bbfb-5b0823062072/volumes"
Nov 01 09:30:22.440881 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:22.440848    4931 generic.go:356] "Generic (PLEG): container finished" podID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" containerID="7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6" exitCode=137
Nov 01 09:30:22.441113 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:22.440912    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-8eae4b"
Nov 01 09:30:22.441113 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:22.440951    4931 scope.go:122] "RemoveContainer" containerID="7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6"
Nov 01 09:30:22.450430 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:22.450411    4931 scope.go:122] "RemoveContainer" containerID="7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6"
Nov 01 09:30:22.450744 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:22.450724    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6\": container with ID starting with 7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6 not found: ID does not exist" containerID="7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6"
Nov 01 09:30:22.450789 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:22.450755    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6"} err="failed to get container status \"7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6\": rpc error: code = NotFound desc = could not find container \"7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6\": container with ID starting with 7ade07dcbf40d1522fffef6bccf41f1eadfb9b5164547888f6e2d8067b1acaa6 not found: ID does not exist"
Nov 01 09:30:31.022701 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:30:31.022661    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:30:32.390998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:32.390944    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"]
Nov 01 09:30:47.412668 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.412607    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86" podUID="b6e91af4-7ae5-487f-9244-80238b88034a" containerName="main" containerID="cri-o://ecec6a084686a23ad0496674bf8026cde1678d080139055769a61c14573d6526" gracePeriod=45
Nov 01 09:30:47.640767 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.640730    4931 generic.go:356] "Generic (PLEG): container finished" podID="b6e91af4-7ae5-487f-9244-80238b88034a" containerID="ecec6a084686a23ad0496674bf8026cde1678d080139055769a61c14573d6526" exitCode=0
Nov 01 09:30:47.640984 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.640801    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86" event={"ID":"b6e91af4-7ae5-487f-9244-80238b88034a","Type":"ContainerDied","Data":"ecec6a084686a23ad0496674bf8026cde1678d080139055769a61c14573d6526"}
Nov 01 09:30:47.697327 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.697298    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:30:47.878520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878482    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-tmp-dir\") pod \"b6e91af4-7ae5-487f-9244-80238b88034a\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") "
Nov 01 09:30:47.878520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878525    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-dshm\") pod \"b6e91af4-7ae5-487f-9244-80238b88034a\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") "
Nov 01 09:30:47.878797 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878548    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e91af4-7ae5-487f-9244-80238b88034a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e91af4-7ae5-487f-9244-80238b88034a-tls-certs\") pod \"b6e91af4-7ae5-487f-9244-80238b88034a\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") "
Nov 01 09:30:47.878797 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878626    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-home\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-home\") pod \"b6e91af4-7ae5-487f-9244-80238b88034a\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") "
Nov 01 09:30:47.878797 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878687    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-model-cache\") pod \"b6e91af4-7ae5-487f-9244-80238b88034a\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") "
Nov 01 09:30:47.878797 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878747    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e91af4-7ae5-487f-9244-80238b88034a-kube-api-access-wslpl\" (UniqueName: \"kubernetes.io/projected/b6e91af4-7ae5-487f-9244-80238b88034a-kube-api-access-wslpl\") pod \"b6e91af4-7ae5-487f-9244-80238b88034a\" (UID: \"b6e91af4-7ae5-487f-9244-80238b88034a\") "
Nov 01 09:30:47.879008 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878817    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-tmp-dir" pod "b6e91af4-7ae5-487f-9244-80238b88034a" (UID: "b6e91af4-7ae5-487f-9244-80238b88034a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:47.879008 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878931    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-home" pod "b6e91af4-7ae5-487f-9244-80238b88034a" (UID: "b6e91af4-7ae5-487f-9244-80238b88034a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:47.879008 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.878978    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-model-cache" pod "b6e91af4-7ae5-487f-9244-80238b88034a" (UID: "b6e91af4-7ae5-487f-9244-80238b88034a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:47.879258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.879239    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:47.879258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.879258    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:47.879374 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.879267    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:47.881359 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.881333    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e91af4-7ae5-487f-9244-80238b88034a-kube-api-access-wslpl" pod "b6e91af4-7ae5-487f-9244-80238b88034a" (UID: "b6e91af4-7ae5-487f-9244-80238b88034a"). InnerVolumeSpecName "kube-api-access-wslpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:30:47.881475 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.881361    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-dshm" pod "b6e91af4-7ae5-487f-9244-80238b88034a" (UID: "b6e91af4-7ae5-487f-9244-80238b88034a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:30:47.881475 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.881433    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e91af4-7ae5-487f-9244-80238b88034a-tls-certs" pod "b6e91af4-7ae5-487f-9244-80238b88034a" (UID: "b6e91af4-7ae5-487f-9244-80238b88034a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:30:47.979998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.979917    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wslpl\" (UniqueName: \"kubernetes.io/projected/b6e91af4-7ae5-487f-9244-80238b88034a-kube-api-access-wslpl\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:47.979998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.979948    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e91af4-7ae5-487f-9244-80238b88034a-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:47.979998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:47.979959    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e91af4-7ae5-487f-9244-80238b88034a-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:30:48.599835 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:48.599788    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86" podUID="b6e91af4-7ae5-487f-9244-80238b88034a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:30:48.648560 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:48.648526    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"
Nov 01 09:30:48.648560 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:48.648537    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86" event={"ID":"b6e91af4-7ae5-487f-9244-80238b88034a","Type":"ContainerDied","Data":"801a4cd26c512f63df6691353e0714550a60e6661520353a4d7780b7b7b12192"}
Nov 01 09:30:48.648804 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:48.648593    4931 scope.go:122] "RemoveContainer" containerID="ecec6a084686a23ad0496674bf8026cde1678d080139055769a61c14573d6526"
Nov 01 09:30:48.809232 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:48.809194    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"]
Nov 01 09:30:48.814358 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:48.814325    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a87a1d/auth-test-model-a87a1d-kserve-5dcf84c769-cgs86"]
Nov 01 09:30:50.049865 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:30:50.049819    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e91af4-7ae5-487f-9244-80238b88034a" path="/var/lib/kubelet/pods/b6e91af4-7ae5-487f-9244-80238b88034a/volumes"
Nov 01 09:31:01.023010 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:31:01.022971    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:31:13.288707 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.288666    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"]
Nov 01 09:31:13.289210 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289194    4931 cpu_manager.go:414] "containerMap: removing container" podUID="b6e91af4-7ae5-487f-9244-80238b88034a" containerName="main"
Nov 01 09:31:13.289262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289211    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e91af4-7ae5-487f-9244-80238b88034a" containerName="main"
Nov 01 09:31:13.289262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289222    4931 cpu_manager.go:414] "containerMap: removing container" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.289262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289227    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.289262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289239    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.289262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289244    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.289262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289251    4931 cpu_manager.go:414] "containerMap: removing container" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.289262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289255    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.289262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289263    4931 cpu_manager.go:414] "containerMap: removing container" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289268    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289288    4931 cpu_manager.go:414] "containerMap: removing container" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289295    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289307    4931 cpu_manager.go:414] "containerMap: removing container" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289311    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289318    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289322    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289329    4931 cpu_manager.go:414] "containerMap: removing container" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" containerName="test-curl-7415-8eae4b"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289335    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" containerName="test-curl-7415-8eae4b"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289347    4931 cpu_manager.go:414] "containerMap: removing container" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289353    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289366    4931 cpu_manager.go:414] "containerMap: removing container" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289505    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289521    4931 cpu_manager.go:414] "containerMap: removing container" podUID="824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" containerName="main"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289529    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" containerName="main"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289546    4931 cpu_manager.go:414] "containerMap: removing container" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289551    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289560    4931 cpu_manager.go:414] "containerMap: removing container" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.289583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289565    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289772    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289788    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289797    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289807    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289818    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="824c6bdd-5ccc-4e8b-89b7-b23d901b36f0" containerName="main"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289830    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b397369-7fc2-408f-bbfb-5b0823062072" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289836    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289841    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289846    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289850    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e91af4-7ae5-487f-9244-80238b88034a" containerName="main"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289857    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289862    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289867    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.290259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.289876    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0a310c-67dd-4274-abb5-c08ce59f49a1" containerName="test-curl-7415-8eae4b"
Nov 01 09:31:13.293322 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.293292    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:13.346395 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.346352    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"]
Nov 01 09:31:13.389113 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.382849    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbb6ad9-8f83-4d91-b360-686609d42899-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf\" (UID: \"efbb6ad9-8f83-4d91-b360-686609d42899\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:13.389587 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.389497    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvgp\" (UniqueName: \"kubernetes.io/projected/efbb6ad9-8f83-4d91-b360-686609d42899-kube-api-access-nmvgp\") pod \"payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf\" (UID: \"efbb6ad9-8f83-4d91-b360-686609d42899\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:13.412834 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.400301    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e3728d-dockercfg-dnvgw\""
Nov 01 09:31:13.412834 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.400437    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e3728d\""
Nov 01 09:31:13.451629 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.451598    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"]
Nov 01 09:31:13.452208 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452193    4931 cpu_manager.go:414] "containerMap: removing container" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.452269 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452212    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.452269 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452227    4931 cpu_manager.go:414] "containerMap: removing container" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.452269 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452233    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.452269 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452243    4931 cpu_manager.go:414] "containerMap: removing container" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.452269 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452247    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.452269 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452256    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.452269 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452260    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.452489 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452337    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7cf5b1-5eea-43f7-addf-92bd1f2abe93" containerName="cleanup"
Nov 01 09:31:13.452489 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452347    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75606ff-63bb-4cfc-9c68-1c223b40115e" containerName="cleanup"
Nov 01 09:31:13.452489 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452362    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc863aca-8a42-40ce-85d0-d36e8a1d4eb2" containerName="cleanup"
Nov 01 09:31:13.452489 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.452370    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d5adfd-0c38-4e30-ab0c-f5de89a57121" containerName="cleanup"
Nov 01 09:31:13.455569 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.455552    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:13.491552 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.491516    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbb6ad9-8f83-4d91-b360-686609d42899-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf\" (UID: \"efbb6ad9-8f83-4d91-b360-686609d42899\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:13.491737 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.491581    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvgp\" (UniqueName: \"kubernetes.io/projected/efbb6ad9-8f83-4d91-b360-686609d42899-kube-api-access-nmvgp\") pod \"payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf\" (UID: \"efbb6ad9-8f83-4d91-b360-686609d42899\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:13.492234 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.492213    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbb6ad9-8f83-4d91-b360-686609d42899-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf\" (UID: \"efbb6ad9-8f83-4d91-b360-686609d42899\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:13.509899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.509860    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"]
Nov 01 09:31:13.550868 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.550804    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvgp\" (UniqueName: \"kubernetes.io/projected/efbb6ad9-8f83-4d91-b360-686609d42899-kube-api-access-nmvgp\") pod \"payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf\" (UID: \"efbb6ad9-8f83-4d91-b360-686609d42899\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:13.592958 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.592924    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9qc\" (UniqueName: \"kubernetes.io/projected/c0e83371-c3ed-435e-8716-c2cfc2922da0-kube-api-access-5l9qc\") pod \"payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4\" (UID: \"c0e83371-c3ed-435e-8716-c2cfc2922da0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:13.592958 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.592968    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e83371-c3ed-435e-8716-c2cfc2922da0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4\" (UID: \"c0e83371-c3ed-435e-8716-c2cfc2922da0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:13.607306 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.607276    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:13.694624 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.694587    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9qc\" (UniqueName: \"kubernetes.io/projected/c0e83371-c3ed-435e-8716-c2cfc2922da0-kube-api-access-5l9qc\") pod \"payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4\" (UID: \"c0e83371-c3ed-435e-8716-c2cfc2922da0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:13.694624 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.694627    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e83371-c3ed-435e-8716-c2cfc2922da0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4\" (UID: \"c0e83371-c3ed-435e-8716-c2cfc2922da0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:13.695899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.695877    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e83371-c3ed-435e-8716-c2cfc2922da0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4\" (UID: \"c0e83371-c3ed-435e-8716-c2cfc2922da0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:13.753838 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.753803    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9qc\" (UniqueName: \"kubernetes.io/projected/c0e83371-c3ed-435e-8716-c2cfc2922da0-kube-api-access-5l9qc\") pod \"payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4\" (UID: \"c0e83371-c3ed-435e-8716-c2cfc2922da0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:13.769300 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.769272    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:13.984037 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:13.983994    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"]
Nov 01 09:31:13.985171 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:31:13.985143    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbb6ad9_8f83_4d91_b360_686609d42899.slice/crio-781ab527b1745f2d05de5e6fe21ed2510b56706303f9dc004b64ea8333638eb6 WatchSource:0}: Error finding container 781ab527b1745f2d05de5e6fe21ed2510b56706303f9dc004b64ea8333638eb6: Status 404 returned error can't find the container with id 781ab527b1745f2d05de5e6fe21ed2510b56706303f9dc004b64ea8333638eb6
Nov 01 09:31:14.143653 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:14.143625    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"]
Nov 01 09:31:14.144642 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:31:14.144618    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e83371_c3ed_435e_8716_c2cfc2922da0.slice/crio-2853bf44c97507fa153f810b920020c9470971e29789828bff26450585db42fa WatchSource:0}: Error finding container 2853bf44c97507fa153f810b920020c9470971e29789828bff26450585db42fa: Status 404 returned error can't find the container with id 2853bf44c97507fa153f810b920020c9470971e29789828bff26450585db42fa
Nov 01 09:31:14.865212 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:14.865177    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf" event={"ID":"efbb6ad9-8f83-4d91-b360-686609d42899","Type":"ContainerStarted","Data":"c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829"}
Nov 01 09:31:14.865669 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:14.865230    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf" event={"ID":"efbb6ad9-8f83-4d91-b360-686609d42899","Type":"ContainerStarted","Data":"781ab527b1745f2d05de5e6fe21ed2510b56706303f9dc004b64ea8333638eb6"}
Nov 01 09:31:14.865669 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:14.865281    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:14.866828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:14.866803    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4" event={"ID":"c0e83371-c3ed-435e-8716-c2cfc2922da0","Type":"ContainerStarted","Data":"acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419"}
Nov 01 09:31:14.866905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:14.866848    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4" event={"ID":"c0e83371-c3ed-435e-8716-c2cfc2922da0","Type":"ContainerStarted","Data":"2853bf44c97507fa153f810b920020c9470971e29789828bff26450585db42fa"}
Nov 01 09:31:14.866951 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:14.866930    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:14.978190 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:14.978129    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf" podStartSLOduration=1.978114259 podStartE2EDuration="1.978114259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:31:14.976136591 +0000 UTC m=+3545.091009560" watchObservedRunningTime="2026-11-01 09:31:14.978114259 +0000 UTC m=+3545.092987260"
Nov 01 09:31:15.094303 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:15.093310    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4" podStartSLOduration=2.093285635 podStartE2EDuration="2.093285635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:31:15.087108068 +0000 UTC m=+3545.201981036" watchObservedRunningTime="2026-11-01 09:31:15.093285635 +0000 UTC m=+3545.208158605"
Nov 01 09:31:25.876643 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:25.876556    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:31:25.877018 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:25.876686    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:31:31.024689 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:31:31.024647    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:31:42.946296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:42.946255    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"]
Nov 01 09:31:42.950413 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:42.950390    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.000560 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.000523    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-49e170-serving-cert\""
Nov 01 09:31:43.051220 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.051139    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxvp\" (UniqueName: \"kubernetes.io/projected/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-kube-api-access-zrxvp\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.051535 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.051511    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.051798 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.051775    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.051955 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.051939    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-metrics-certs\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.054413 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.054391    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"]
Nov 01 09:31:43.152898 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.152864    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.153155 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.152927    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-metrics-certs\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.153155 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:31:43.153112    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-serving-cert: secret "maas-api-e2e-extmod-a-49e170-serving-cert" not found
Nov 01 09:31:43.153296 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:31:43.153191    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls podName:064d83c4-5e8f-4aec-9d64-6c4b89453c8d nodeName:}" failed. No retries permitted until 2026-11-01 09:31:43.65317467 +0000 UTC m=+3573.768047619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls") pod "maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz" (UID: "064d83c4-5e8f-4aec-9d64-6c4b89453c8d") : secret "maas-api-e2e-extmod-a-49e170-serving-cert" not found
Nov 01 09:31:43.153296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.153179    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxvp\" (UniqueName: \"kubernetes.io/projected/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-kube-api-access-zrxvp\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.153296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.153231    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.153931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.153906    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.155714 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.155695    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-metrics-certs\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.212500 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.212416    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxvp\" (UniqueName: \"kubernetes.io/projected/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-kube-api-access-zrxvp\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.659738 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.659699    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.662598 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.662572    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:43.865103 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:43.865067    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:44.243490 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:44.243459    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"]
Nov 01 09:31:44.244639 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:31:44.244612    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064d83c4_5e8f_4aec_9d64_6c4b89453c8d.slice/crio-da6a6003fbcdd68858d2c1b978a275887e12bc6ba4e83d9fb71b6817f5e73fd3 WatchSource:0}: Error finding container da6a6003fbcdd68858d2c1b978a275887e12bc6ba4e83d9fb71b6817f5e73fd3: Status 404 returned error can't find the container with id da6a6003fbcdd68858d2c1b978a275887e12bc6ba4e83d9fb71b6817f5e73fd3
Nov 01 09:31:45.088329 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:45.088293    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz" event={"ID":"064d83c4-5e8f-4aec-9d64-6c4b89453c8d","Type":"ContainerStarted","Data":"da6a6003fbcdd68858d2c1b978a275887e12bc6ba4e83d9fb71b6817f5e73fd3"}
Nov 01 09:31:46.097318 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:46.097282    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz" event={"ID":"064d83c4-5e8f-4aec-9d64-6c4b89453c8d","Type":"ContainerStarted","Data":"886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959"}
Nov 01 09:31:46.097843 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:46.097504    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:31:46.252128 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:46.252073    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz" podStartSLOduration=3.3753113040000002 podStartE2EDuration="4.252058472s" totalImagesPullingTime="876.747168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:31:46.249985188 +0000 UTC m=+3576.364858156" watchObservedRunningTime="2026-11-01 09:31:46.252058472 +0000 UTC m=+3576.366931440"
Nov 01 09:31:52.111220 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:31:52.111192    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:32:01.024419 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:32:01.024380    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:32:10.695204 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:10.695170    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:32:10.704958 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:10.704926    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:32:10.727437 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:10.727408    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:32:10.731088 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:10.730999    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:32:12.141954 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.141915    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"]
Nov 01 09:32:12.145935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.145912    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.197332 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.197300    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-25d9fe-serving-cert\""
Nov 01 09:32:12.201441 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.201410    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"]
Nov 01 09:32:12.297058 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.296971    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6155ba00-1907-489f-b77e-6b5264b8fac5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.297264 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.297127    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.297264 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.297162    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6vr\" (UniqueName: \"kubernetes.io/projected/6155ba00-1907-489f-b77e-6b5264b8fac5-kube-api-access-qm6vr\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.297264 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.297223    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-metrics-certs\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.399351 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.399235    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6155ba00-1907-489f-b77e-6b5264b8fac5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.399544 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.399363    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.399544 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.399398    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6vr\" (UniqueName: \"kubernetes.io/projected/6155ba00-1907-489f-b77e-6b5264b8fac5-kube-api-access-qm6vr\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.399544 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.399482    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-metrics-certs\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.400274 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.400247    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6155ba00-1907-489f-b77e-6b5264b8fac5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.403233 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.403179    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.403384 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.403241    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-metrics-certs\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.459264 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.459235    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6vr\" (UniqueName: \"kubernetes.io/projected/6155ba00-1907-489f-b77e-6b5264b8fac5-kube-api-access-qm6vr\") pod \"maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.459476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.459452    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:12.837690 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:12.837649    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"]
Nov 01 09:32:12.838677 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:32:12.838652    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6155ba00_1907_489f_b77e_6b5264b8fac5.slice/crio-723540b7156c4c6cb64892420626b0f334783a1d34cda6c5457c6b90d7092fd0 WatchSource:0}: Error finding container 723540b7156c4c6cb64892420626b0f334783a1d34cda6c5457c6b90d7092fd0: Status 404 returned error can't find the container with id 723540b7156c4c6cb64892420626b0f334783a1d34cda6c5457c6b90d7092fd0
Nov 01 09:32:13.282279 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:13.282242    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs" event={"ID":"6155ba00-1907-489f-b77e-6b5264b8fac5","Type":"ContainerStarted","Data":"723540b7156c4c6cb64892420626b0f334783a1d34cda6c5457c6b90d7092fd0"}
Nov 01 09:32:14.291310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:14.291266    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs" event={"ID":"6155ba00-1907-489f-b77e-6b5264b8fac5","Type":"ContainerStarted","Data":"896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a"}
Nov 01 09:32:14.291766 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:14.291485    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:14.410075 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:14.406558    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs" podStartSLOduration=1.4815025130000001 podStartE2EDuration="2.406537522s" totalImagesPullingTime="925.035009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:32:14.405061321 +0000 UTC m=+3604.519934280" watchObservedRunningTime="2026-11-01 09:32:14.406537522 +0000 UTC m=+3604.521410491"
Nov 01 09:32:20.303314 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:20.303284    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:32:31.023270 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:32:31.023218    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:32:36.238575 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.238540    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"]
Nov 01 09:32:36.244077 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.244051    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.294322 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.294285    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e3728d-serving-cert\""
Nov 01 09:32:36.347018 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.346985    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"]
Nov 01 09:32:36.395719 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.395672    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47a72b8-69f2-4033-8160-cd8180fd776a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.395914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.395855    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-metrics-certs\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.396065 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.396045    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.396163 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.396124    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5zm\" (UniqueName: \"kubernetes.io/projected/f47a72b8-69f2-4033-8160-cd8180fd776a-kube-api-access-4n5zm\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.502896 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.502802    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.503472 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:32:36.503270    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-serving-cert: secret "maas-api-e2e-extmod-a-e3728d-serving-cert" not found
Nov 01 09:32:36.503621 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:32:36.503553    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls podName:f47a72b8-69f2-4033-8160-cd8180fd776a nodeName:}" failed. No retries permitted until 2026-11-01 09:32:37.00352813 +0000 UTC m=+3627.118401094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls") pod "maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds" (UID: "f47a72b8-69f2-4033-8160-cd8180fd776a") : secret "maas-api-e2e-extmod-a-e3728d-serving-cert" not found
Nov 01 09:32:36.503621 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.503583    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5zm\" (UniqueName: \"kubernetes.io/projected/f47a72b8-69f2-4033-8160-cd8180fd776a-kube-api-access-4n5zm\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.503746 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.503666    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47a72b8-69f2-4033-8160-cd8180fd776a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.503809 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.503777    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-metrics-certs\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.504730 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.504684    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47a72b8-69f2-4033-8160-cd8180fd776a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.506969 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.506945    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-metrics-certs\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:36.565088 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:36.565054    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5zm\" (UniqueName: \"kubernetes.io/projected/f47a72b8-69f2-4033-8160-cd8180fd776a-kube-api-access-4n5zm\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:37.010058 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:37.010007    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:37.012935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:37.012888    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:37.158102 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:37.158063    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:37.542653 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:37.542612    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"]
Nov 01 09:32:37.543806 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:32:37.543775    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47a72b8_69f2_4033_8160_cd8180fd776a.slice/crio-6c0b943d18d6d2f14ca7db823970f0aa917a369bffb320f39aa729a81e9a8de9 WatchSource:0}: Error finding container 6c0b943d18d6d2f14ca7db823970f0aa917a369bffb320f39aa729a81e9a8de9: Status 404 returned error can't find the container with id 6c0b943d18d6d2f14ca7db823970f0aa917a369bffb320f39aa729a81e9a8de9
Nov 01 09:32:38.465238 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:38.465199    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds" event={"ID":"f47a72b8-69f2-4033-8160-cd8180fd776a","Type":"ContainerStarted","Data":"6c0b943d18d6d2f14ca7db823970f0aa917a369bffb320f39aa729a81e9a8de9"}
Nov 01 09:32:39.475734 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:39.475698    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds" event={"ID":"f47a72b8-69f2-4033-8160-cd8180fd776a","Type":"ContainerStarted","Data":"078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38"}
Nov 01 09:32:39.476236 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:39.475818    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:32:39.590527 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:39.590468    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds" podStartSLOduration=2.770224252 podStartE2EDuration="3.590453651s" totalImagesPullingTime="820.229399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:32:39.588397853 +0000 UTC m=+3629.703270833" watchObservedRunningTime="2026-11-01 09:32:39.590453651 +0000 UTC m=+3629.705326619"
Nov 01 09:32:45.491798 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:32:45.491768    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:33:01.024504 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:33:01.024465    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:33:07.088090 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:07.088053    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7406-9ba35d"]
Nov 01 09:33:07.092430 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:07.092400    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:07.142899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:07.142866    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-44wvn\""
Nov 01 09:33:07.144819 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:07.144793    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7406-9ba35d"]
Nov 01 09:33:07.281243 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:07.281204    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jjv\" (UniqueName: \"kubernetes.io/projected/0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a-kube-api-access-v9jjv\") pod \"test-curl-7406-9ba35d\" (UID: \"0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a\") " pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:07.382910 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:07.382853    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jjv\" (UniqueName: \"kubernetes.io/projected/0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a-kube-api-access-v9jjv\") pod \"test-curl-7406-9ba35d\" (UID: \"0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a\") " pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:07.442114 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:07.442077    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jjv\" (UniqueName: \"kubernetes.io/projected/0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a-kube-api-access-v9jjv\") pod \"test-curl-7406-9ba35d\" (UID: \"0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a\") " pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:07.704703 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:07.704620    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:08.080227 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:08.080175    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7406-9ba35d"]
Nov 01 09:33:08.094102 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:33:08.093915    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ef9c942_35c4_4fde_82a5_4b1ad1dc9d5a.slice/crio-f4e437bc34f63274fb3ac4c4e38cbd45494869ef5701600b573471e2766a510c WatchSource:0}: Error finding container f4e437bc34f63274fb3ac4c4e38cbd45494869ef5701600b573471e2766a510c: Status 404 returned error can't find the container with id f4e437bc34f63274fb3ac4c4e38cbd45494869ef5701600b573471e2766a510c
Nov 01 09:33:08.673448 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:08.673403    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7406-9ba35d" event={"ID":"0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a","Type":"ContainerStarted","Data":"ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed"}
Nov 01 09:33:08.673664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:08.673454    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7406-9ba35d" event={"ID":"0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a","Type":"ContainerStarted","Data":"f4e437bc34f63274fb3ac4c4e38cbd45494869ef5701600b573471e2766a510c"}
Nov 01 09:33:08.782653 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:08.782599    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7406-9ba35d" podStartSLOduration=1.782584325 podStartE2EDuration="1.782584325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:08.781392439 +0000 UTC m=+3658.896265408" watchObservedRunningTime="2026-11-01 09:33:08.782584325 +0000 UTC m=+3658.897457292"
Nov 01 09:33:10.223996 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:10.223964    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7406-9ba35d"]
Nov 01 09:33:10.229096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:10.229061    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7406-9ba35d"]
Nov 01 09:33:10.689401 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:10.689351    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7406-9ba35d" podUID="0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" containerName="test-curl-7406-9ba35d" containerID="cri-o://ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed" gracePeriod=2
Nov 01 09:33:10.738726 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:33:10.738686    4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7406-9ba35d\" is forbidden: User \"system:node:ip-10-0-143-244.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.us-west-2.compute.internal' and this object" podUID="0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:12.944992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:12.944915    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:13.051363 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:13.051333    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a-kube-api-access-v9jjv\" (UniqueName: \"kubernetes.io/projected/0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a-kube-api-access-v9jjv\") pod \"0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a\" (UID: \"0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a\") "
Nov 01 09:33:13.053610 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:13.053579    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a-kube-api-access-v9jjv" pod "0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" (UID: "0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a"). InnerVolumeSpecName "kube-api-access-v9jjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:33:13.153141 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:13.153109    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9jjv\" (UniqueName: \"kubernetes.io/projected/0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a-kube-api-access-v9jjv\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:33:13.714107 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:13.714068    4931 generic.go:356] "Generic (PLEG): container finished" podID="0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" containerID="ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed" exitCode=137
Nov 01 09:33:13.714307 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:13.714116    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:13.714307 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:13.714177    4931 scope.go:122] "RemoveContainer" containerID="ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed"
Nov 01 09:33:13.724474 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:13.724456    4931 scope.go:122] "RemoveContainer" containerID="ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed"
Nov 01 09:33:13.724776 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:33:13.724756    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed\": container with ID starting with ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed not found: ID does not exist" containerID="ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed"
Nov 01 09:33:13.724844 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:13.724785    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed"} err="failed to get container status \"ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed\": rpc error: code = NotFound desc = could not find container \"ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed\": container with ID starting with ff90b4f7c23cfcd40a689073c16b083d4f86283ef694df01c90317e818d3f6ed not found: ID does not exist"
Nov 01 09:33:13.771833 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:33:13.771798    4931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7406-9ba35d\" is forbidden: User \"system:node:ip-10-0-143-244.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.us-west-2.compute.internal' and this object" podUID="0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" pod="opendatahub/test-curl-7406-9ba35d"
Nov 01 09:33:14.048284 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:33:14.048207    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" path="/var/lib/kubelet/pods/0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a/volumes"
Nov 01 09:33:31.023372 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:33:31.023336    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:34:01.022934 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:34:01.022894    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:34:14.618058 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:14.617992    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"]
Nov 01 09:34:14.725260 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:14.725225    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"]
Nov 01 09:34:19.618956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:19.618807    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4" podUID="c0e83371-c3ed-435e-8716-c2cfc2922da0" containerName="payload-processing" containerID="cri-o://acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419" gracePeriod=25
Nov 01 09:34:19.725778 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:19.725706    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf" podUID="efbb6ad9-8f83-4d91-b360-686609d42899" containerName="payload-pre-processing" containerID="cri-o://c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829" gracePeriod=25
Nov 01 09:34:19.883375 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:19.883298    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:34:20.004775 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.004739    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:34:20.007859 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.007821    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e83371-c3ed-435e-8716-c2cfc2922da0-kube-api-access-5l9qc\" (UniqueName: \"kubernetes.io/projected/c0e83371-c3ed-435e-8716-c2cfc2922da0-kube-api-access-5l9qc\") pod \"c0e83371-c3ed-435e-8716-c2cfc2922da0\" (UID: \"c0e83371-c3ed-435e-8716-c2cfc2922da0\") "
Nov 01 09:34:20.008071 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.007888    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e83371-c3ed-435e-8716-c2cfc2922da0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e83371-c3ed-435e-8716-c2cfc2922da0-plugins-config-volume\") pod \"c0e83371-c3ed-435e-8716-c2cfc2922da0\" (UID: \"c0e83371-c3ed-435e-8716-c2cfc2922da0\") "
Nov 01 09:34:20.008354 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.008324    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e83371-c3ed-435e-8716-c2cfc2922da0-plugins-config-volume" pod "c0e83371-c3ed-435e-8716-c2cfc2922da0" (UID: "c0e83371-c3ed-435e-8716-c2cfc2922da0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:34:20.008536 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.008512    4931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e83371-c3ed-435e-8716-c2cfc2922da0-plugins-config-volume\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:20.011405 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.011369    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e83371-c3ed-435e-8716-c2cfc2922da0-kube-api-access-5l9qc" pod "c0e83371-c3ed-435e-8716-c2cfc2922da0" (UID: "c0e83371-c3ed-435e-8716-c2cfc2922da0"). InnerVolumeSpecName "kube-api-access-5l9qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:34:20.109483 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.109442    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbb6ad9-8f83-4d91-b360-686609d42899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbb6ad9-8f83-4d91-b360-686609d42899-plugins-config-volume\") pod \"efbb6ad9-8f83-4d91-b360-686609d42899\" (UID: \"efbb6ad9-8f83-4d91-b360-686609d42899\") "
Nov 01 09:34:20.109688 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.109518    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbb6ad9-8f83-4d91-b360-686609d42899-kube-api-access-nmvgp\" (UniqueName: \"kubernetes.io/projected/efbb6ad9-8f83-4d91-b360-686609d42899-kube-api-access-nmvgp\") pod \"efbb6ad9-8f83-4d91-b360-686609d42899\" (UID: \"efbb6ad9-8f83-4d91-b360-686609d42899\") "
Nov 01 09:34:20.109922 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.109876    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9qc\" (UniqueName: \"kubernetes.io/projected/c0e83371-c3ed-435e-8716-c2cfc2922da0-kube-api-access-5l9qc\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:20.109922 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.109870    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbb6ad9-8f83-4d91-b360-686609d42899-plugins-config-volume" pod "efbb6ad9-8f83-4d91-b360-686609d42899" (UID: "efbb6ad9-8f83-4d91-b360-686609d42899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:34:20.111867 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.111843    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbb6ad9-8f83-4d91-b360-686609d42899-kube-api-access-nmvgp" pod "efbb6ad9-8f83-4d91-b360-686609d42899" (UID: "efbb6ad9-8f83-4d91-b360-686609d42899"). InnerVolumeSpecName "kube-api-access-nmvgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:34:20.204156 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.204064    4931 generic.go:356] "Generic (PLEG): container finished" podID="efbb6ad9-8f83-4d91-b360-686609d42899" containerID="c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829" exitCode=0
Nov 01 09:34:20.204323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.204156    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"
Nov 01 09:34:20.204323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.204160    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf" event={"ID":"efbb6ad9-8f83-4d91-b360-686609d42899","Type":"ContainerDied","Data":"c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829"}
Nov 01 09:34:20.204323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.204242    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf" event={"ID":"efbb6ad9-8f83-4d91-b360-686609d42899","Type":"ContainerDied","Data":"781ab527b1745f2d05de5e6fe21ed2510b56706303f9dc004b64ea8333638eb6"}
Nov 01 09:34:20.204323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.204288    4931 scope.go:122] "RemoveContainer" containerID="c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829"
Nov 01 09:34:20.205587 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.205562    4931 generic.go:356] "Generic (PLEG): container finished" podID="c0e83371-c3ed-435e-8716-c2cfc2922da0" containerID="acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419" exitCode=0
Nov 01 09:34:20.205708 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.205606    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"
Nov 01 09:34:20.205708 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.205649    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4" event={"ID":"c0e83371-c3ed-435e-8716-c2cfc2922da0","Type":"ContainerDied","Data":"acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419"}
Nov 01 09:34:20.205708 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.205705    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4" event={"ID":"c0e83371-c3ed-435e-8716-c2cfc2922da0","Type":"ContainerDied","Data":"2853bf44c97507fa153f810b920020c9470971e29789828bff26450585db42fa"}
Nov 01 09:34:20.210702 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.210673    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmvgp\" (UniqueName: \"kubernetes.io/projected/efbb6ad9-8f83-4d91-b360-686609d42899-kube-api-access-nmvgp\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:20.210845 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.210705    4931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbb6ad9-8f83-4d91-b360-686609d42899-plugins-config-volume\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:34:20.215366 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.215348    4931 scope.go:122] "RemoveContainer" containerID="c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829"
Nov 01 09:34:20.215675 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:34:20.215654    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829\": container with ID starting with c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829 not found: ID does not exist" containerID="c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829"
Nov 01 09:34:20.215723 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.215688    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829"} err="failed to get container status \"c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829\": rpc error: code = NotFound desc = could not find container \"c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829\": container with ID starting with c148fcf14a873a7d5289acaa949f78d2787a387c76579563ad490a4a4ba72829 not found: ID does not exist"
Nov 01 09:34:20.215723 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.215712    4931 scope.go:122] "RemoveContainer" containerID="acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419"
Nov 01 09:34:20.225726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.225699    4931 scope.go:122] "RemoveContainer" containerID="acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419"
Nov 01 09:34:20.226050 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:34:20.226009    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419\": container with ID starting with acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419 not found: ID does not exist" containerID="acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419"
Nov 01 09:34:20.226127 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.226043    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419"} err="failed to get container status \"acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419\": rpc error: code = NotFound desc = could not find container \"acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419\": container with ID starting with acc0abd19d92b9ee4ad249d5aa1d7062d5d00eef938556ae74f997a115bfd419 not found: ID does not exist"
Nov 01 09:34:20.362446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.362411    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"]
Nov 01 09:34:20.368857 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.368827    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e3728d-5f8d895485-cfbp4"]
Nov 01 09:34:20.473331 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.473243    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"]
Nov 01 09:34:20.479338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:20.479300    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e3728d-6c858c747f-cxxmf"]
Nov 01 09:34:22.049408 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:22.049377    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e83371-c3ed-435e-8716-c2cfc2922da0" path="/var/lib/kubelet/pods/c0e83371-c3ed-435e-8716-c2cfc2922da0/volumes"
Nov 01 09:34:22.049845 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:34:22.049831    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbb6ad9-8f83-4d91-b360-686609d42899" path="/var/lib/kubelet/pods/efbb6ad9-8f83-4d91-b360-686609d42899/volumes"
Nov 01 09:34:31.023929 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:34:31.023887    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:35:01.023101 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:35:01.023059    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:35:04.537065 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537035    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"]
Nov 01 09:35:04.537558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537545    4931 cpu_manager.go:414] "containerMap: removing container" podUID="0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" containerName="test-curl-7406-9ba35d"
Nov 01 09:35:04.537603 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537559    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" containerName="test-curl-7406-9ba35d"
Nov 01 09:35:04.537603 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537581    4931 cpu_manager.go:414] "containerMap: removing container" podUID="c0e83371-c3ed-435e-8716-c2cfc2922da0" containerName="payload-processing"
Nov 01 09:35:04.537603 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537586    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e83371-c3ed-435e-8716-c2cfc2922da0" containerName="payload-processing"
Nov 01 09:35:04.537603 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537601    4931 cpu_manager.go:414] "containerMap: removing container" podUID="efbb6ad9-8f83-4d91-b360-686609d42899" containerName="payload-pre-processing"
Nov 01 09:35:04.537729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537606    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbb6ad9-8f83-4d91-b360-686609d42899" containerName="payload-pre-processing"
Nov 01 09:35:04.537729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537696    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef9c942-35c4-4fde-82a5-4b1ad1dc9d5a" containerName="test-curl-7406-9ba35d"
Nov 01 09:35:04.537729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537706    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e83371-c3ed-435e-8716-c2cfc2922da0" containerName="payload-processing"
Nov 01 09:35:04.537729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.537714    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbb6ad9-8f83-4d91-b360-686609d42899" containerName="payload-pre-processing"
Nov 01 09:35:04.541172 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.541157    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.598207 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.598173    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"]
Nov 01 09:35:04.640692 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.640657    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b0d0f3-serving-cert\""
Nov 01 09:35:04.647399 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.647368    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.647542 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.647403    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.647542 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.647424    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhcq\" (UniqueName: \"kubernetes.io/projected/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-kube-api-access-flhcq\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.647668 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.647570    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-metrics-certs\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.749417 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.749383    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.749417 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.749423    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.749671 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.749443    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhcq\" (UniqueName: \"kubernetes.io/projected/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-kube-api-access-flhcq\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.749671 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.749478    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-metrics-certs\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.749671 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:35:04.749531    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-serving-cert: secret "maas-api-e2e-extmod-b-b0d0f3-serving-cert" not found
Nov 01 09:35:04.749671 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:35:04.749628    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls podName:cdf4d3e3-c5c6-4c54-8b2e-6d693c144737 nodeName:}" failed. No retries permitted until 2026-11-01 09:35:05.249605295 +0000 UTC m=+3775.364478254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls") pod "maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6" (UID: "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737") : secret "maas-api-e2e-extmod-b-b0d0f3-serving-cert" not found
Nov 01 09:35:04.750113 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.750092    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.752220 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.752189    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-metrics-certs\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:04.811140 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:04.811071    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhcq\" (UniqueName: \"kubernetes.io/projected/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-kube-api-access-flhcq\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:05.257078 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:05.257042    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:05.259793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:05.259760    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:05.455055 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:05.455000    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:05.828815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:05.828787    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"]
Nov 01 09:35:05.829704 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:35:05.829682    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf4d3e3_c5c6_4c54_8b2e_6d693c144737.slice/crio-a8a058414a3e2790d11428b0406ce5d8c711f1ee3be5301b885917aad2c9ecd1 WatchSource:0}: Error finding container a8a058414a3e2790d11428b0406ce5d8c711f1ee3be5301b885917aad2c9ecd1: Status 404 returned error can't find the container with id a8a058414a3e2790d11428b0406ce5d8c711f1ee3be5301b885917aad2c9ecd1
Nov 01 09:35:06.532457 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:06.532416    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6" event={"ID":"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737","Type":"ContainerStarted","Data":"a8a058414a3e2790d11428b0406ce5d8c711f1ee3be5301b885917aad2c9ecd1"}
Nov 01 09:35:07.553797 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:07.553759    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6" event={"ID":"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737","Type":"ContainerStarted","Data":"9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a"}
Nov 01 09:35:07.554307 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:07.553826    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:07.667410 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:07.667349    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6" podStartSLOduration=2.739390598 podStartE2EDuration="3.667335073s" totalImagesPullingTime="927.944475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:35:07.666646452 +0000 UTC m=+3777.781519421" watchObservedRunningTime="2026-11-01 09:35:07.667335073 +0000 UTC m=+3777.782208040"
Nov 01 09:35:13.565830 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:13.565794    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:35:19.030379 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.030345    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"]
Nov 01 09:35:19.034392 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.034368    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.085793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.085761    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-1f3121-serving-cert\""
Nov 01 09:35:19.130330 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.130290    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5bt\" (UniqueName: \"kubernetes.io/projected/3defcc7b-114e-4175-98b5-e7ae11495867-kube-api-access-5g5bt\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.130330 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.130332    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.130560 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.130355    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3defcc7b-114e-4175-98b5-e7ae11495867-openshift-service-ca\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.130617 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.130601    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-metrics-certs\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.136873 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.136850    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"]
Nov 01 09:35:19.232186 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.232145    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-metrics-certs\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.232368 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.232241    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5bt\" (UniqueName: \"kubernetes.io/projected/3defcc7b-114e-4175-98b5-e7ae11495867-kube-api-access-5g5bt\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.232368 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.232261    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.232368 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.232278    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3defcc7b-114e-4175-98b5-e7ae11495867-openshift-service-ca\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.232541 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:35:19.232393    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-serving-cert: secret "maas-api-e2e-autores-1f3121-serving-cert" not found
Nov 01 09:35:19.232541 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:35:19.232459    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls podName:3defcc7b-114e-4175-98b5-e7ae11495867 nodeName:}" failed. No retries permitted until 2026-11-01 09:35:19.732444321 +0000 UTC m=+3789.847317280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls") pod "maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz" (UID: "3defcc7b-114e-4175-98b5-e7ae11495867") : secret "maas-api-e2e-autores-1f3121-serving-cert" not found
Nov 01 09:35:19.233070 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.233044    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3defcc7b-114e-4175-98b5-e7ae11495867-openshift-service-ca\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.234919 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.234901    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-metrics-certs\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.292729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.292642    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5bt\" (UniqueName: \"kubernetes.io/projected/3defcc7b-114e-4175-98b5-e7ae11495867-kube-api-access-5g5bt\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.740439 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.740403    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.743164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.743134    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls\") pod \"maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:19.947820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:19.947783    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:20.327039 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:20.326996    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"]
Nov 01 09:35:20.327843 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:35:20.327819    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3defcc7b_114e_4175_98b5_e7ae11495867.slice/crio-73dad07b8681e25b08482eb91d60ae796844c1919290298211a5396e83492bd9 WatchSource:0}: Error finding container 73dad07b8681e25b08482eb91d60ae796844c1919290298211a5396e83492bd9: Status 404 returned error can't find the container with id 73dad07b8681e25b08482eb91d60ae796844c1919290298211a5396e83492bd9
Nov 01 09:35:20.329302 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:20.329287    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:35:20.650016 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:20.649981    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz" event={"ID":"3defcc7b-114e-4175-98b5-e7ae11495867","Type":"ContainerStarted","Data":"73dad07b8681e25b08482eb91d60ae796844c1919290298211a5396e83492bd9"}
Nov 01 09:35:21.658735 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:21.658635    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz" event={"ID":"3defcc7b-114e-4175-98b5-e7ae11495867","Type":"ContainerStarted","Data":"5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0"}
Nov 01 09:35:21.658735 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:21.658705    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:21.778218 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:21.778165    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz" podStartSLOduration=2.726083888 podStartE2EDuration="3.778150427s" totalImagesPullingTime="1.052066539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:35:21.77711929 +0000 UTC m=+3791.891992258" watchObservedRunningTime="2026-11-01 09:35:21.778150427 +0000 UTC m=+3791.893023393"
Nov 01 09:35:27.672583 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:27.672551    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:35:31.023818 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:35:31.023773    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:35:44.363142 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:44.363111    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"]
Nov 01 09:35:44.363725 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:44.363317    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" containerID="cri-o://a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef" gracePeriod=30
Nov 01 09:35:48.686850 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:48.686816    4931 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:35:48.687264 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:48.686875    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:35:49.637110 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.637040    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:35:49.725523 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725486    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-envoy\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725536    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-kube-api-access-hqj4v\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-kube-api-access-hqj4v\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725597    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-data\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725623    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-token\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725719    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-socket\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725754    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-credential-socket\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725788    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-certs\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725822    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-podinfo\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725918    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-data" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:49.725998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.725933    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istiod-ca-cert\") pod \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\" (UID: \"7ee58f10-6561-41fe-bcc2-d25018b3ce66\") "
Nov 01 09:35:49.726516 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.726279    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-credential-socket" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:49.726516 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.726297    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istiod-ca-cert" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:35:49.726516 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.726459    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-socket" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:49.726748 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.726722    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-certs" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:49.726817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.726747    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.726817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.726769    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.726817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.726784    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.726817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.726797    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.727970 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.727948    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-envoy" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:49.728439 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.728422    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-token" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:35:49.728553 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.728539    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-podinfo" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:35:49.728778 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.728758    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-kube-api-access-hqj4v" pod "7ee58f10-6561-41fe-bcc2-d25018b3ce66" (UID: "7ee58f10-6561-41fe-bcc2-d25018b3ce66"). InnerVolumeSpecName "kube-api-access-hqj4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:35:49.828740 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.828702    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.828899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.828744    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqj4v\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-kube-api-access-hqj4v\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.828899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.828763    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.828899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.828776    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee58f10-6561-41fe-bcc2-d25018b3ce66-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.828899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.828791    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee58f10-6561-41fe-bcc2-d25018b3ce66-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:49.855331 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.855298    4931 generic.go:356] "Generic (PLEG): container finished" podID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerID="a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef" exitCode=0
Nov 01 09:35:49.855518 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.855349    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"
Nov 01 09:35:49.855518 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.855380    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" event={"ID":"7ee58f10-6561-41fe-bcc2-d25018b3ce66","Type":"ContainerDied","Data":"a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef"}
Nov 01 09:35:49.855518 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.855469    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b" event={"ID":"7ee58f10-6561-41fe-bcc2-d25018b3ce66","Type":"ContainerDied","Data":"998e938081d992b9f95c5e918c3fc7d95cbcc3ea5334d13e0cff316f8d000502"}
Nov 01 09:35:49.855518 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.855497    4931 scope.go:122] "RemoveContainer" containerID="a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef"
Nov 01 09:35:49.866623 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.866606    4931 scope.go:122] "RemoveContainer" containerID="a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef"
Nov 01 09:35:49.866882 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:35:49.866865    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef\": container with ID starting with a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef not found: ID does not exist" containerID="a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef"
Nov 01 09:35:49.866931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.866892    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef"} err="failed to get container status \"a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef\": rpc error: code = NotFound desc = could not find container \"a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef\": container with ID starting with a16f054d0f8a8ea650589be0b9a6ac42240ba065d3ca3126a7490144991182ef not found: ID does not exist"
Nov 01 09:35:49.942497 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.942400    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"]
Nov 01 09:35:49.942685 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:49.942637    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy" containerID="cri-o://3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b" gracePeriod=30
Nov 01 09:35:50.017417 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:50.017380    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"]
Nov 01 09:35:50.021430 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:50.021402    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-rlp8b"]
Nov 01 09:35:50.048705 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:50.048670    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" path="/var/lib/kubelet/pods/7ee58f10-6561-41fe-bcc2-d25018b3ce66/volumes"
Nov 01 09:35:53.088349 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:53.088316    4931 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:35:53.088737 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:53.088365    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:35:55.221704 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.221625    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:35:55.295854 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.295820    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-data\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.295914    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872aebee-a5d5-4a7e-a884-7cf86ab02777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872aebee-a5d5-4a7e-a884-7cf86ab02777-istiod-ca-cert\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296102 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296050    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-podinfo\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296102 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296091    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-certs\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296184    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-socket\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296222    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-credential-socket\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296246    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-data" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296256    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-kube-api-access-pnxxh\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-kube-api-access-pnxxh\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296402    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-envoy\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296450    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-token\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-token\") pod \"872aebee-a5d5-4a7e-a884-7cf86ab02777\" (UID: \"872aebee-a5d5-4a7e-a884-7cf86ab02777\") "
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296447    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-certs" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296504    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-socket" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:55.296700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296519    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-credential-socket" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:55.297514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.296727    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872aebee-a5d5-4a7e-a884-7cf86ab02777-istiod-ca-cert" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:35:55.297514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.297194    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.297514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.297218    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872aebee-a5d5-4a7e-a884-7cf86ab02777-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.297514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.297236    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.297514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.297250    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.297514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.297264    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.298770 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.298748    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-podinfo" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:35:55.298907 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.298767    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-kube-api-access-pnxxh" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "kube-api-access-pnxxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:35:55.299228 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.299210    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-token" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:35:55.305377 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.305356    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-envoy" pod "872aebee-a5d5-4a7e-a884-7cf86ab02777" (UID: "872aebee-a5d5-4a7e-a884-7cf86ab02777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:35:55.398805 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.398753    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxxh\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-kube-api-access-pnxxh\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.398805 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.398798    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.398805 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.398812    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.399142 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.398827    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872aebee-a5d5-4a7e-a884-7cf86ab02777-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:35:55.897248 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.897209    4931 generic.go:356] "Generic (PLEG): container finished" podID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerID="3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b" exitCode=0
Nov 01 09:35:55.897462 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.897265    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"
Nov 01 09:35:55.897462 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.897300    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" event={"ID":"872aebee-a5d5-4a7e-a884-7cf86ab02777","Type":"ContainerDied","Data":"3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b"}
Nov 01 09:35:55.897462 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.897373    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh" event={"ID":"872aebee-a5d5-4a7e-a884-7cf86ab02777","Type":"ContainerDied","Data":"6395b80b21cf1e98deef7817a78210610e53bbe047e670b7e12afc709b63f2ab"}
Nov 01 09:35:55.897462 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.897403    4931 scope.go:122] "RemoveContainer" containerID="3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b"
Nov 01 09:35:55.908761 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.908743    4931 scope.go:122] "RemoveContainer" containerID="3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b"
Nov 01 09:35:55.909058 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:35:55.909038    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b\": container with ID starting with 3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b not found: ID does not exist" containerID="3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b"
Nov 01 09:35:55.909116 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:55.909068    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b"} err="failed to get container status \"3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b\": rpc error: code = NotFound desc = could not find container \"3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b\": container with ID starting with 3b81d45fbd0e12e283bdd8460ec822924ab3639206e7bd1e6312ec7ea454118b not found: ID does not exist"
Nov 01 09:35:56.058155 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:56.058124    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"]
Nov 01 09:35:56.105446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:56.105417    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-spwdh"]
Nov 01 09:35:58.048530 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:35:58.048496    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" path="/var/lib/kubelet/pods/872aebee-a5d5-4a7e-a884-7cf86ab02777/volumes"
Nov 01 09:36:01.023631 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:36:01.023596    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:36:31.023145 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:36:31.023110    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:36:59.645704 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:36:59.645660    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"]
Nov 01 09:36:59.646271 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:36:59.645951    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs" podUID="6155ba00-1907-489f-b77e-6b5264b8fac5" containerName="maas-api" containerID="cri-o://896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a" gracePeriod=30
Nov 01 09:36:59.943795 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:36:59.943765    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:37:00.053598 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.053563    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-maas-api-tls\") pod \"6155ba00-1907-489f-b77e-6b5264b8fac5\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") "
Nov 01 09:37:00.053772 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.053730    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6155ba00-1907-489f-b77e-6b5264b8fac5-kube-api-access-qm6vr\" (UniqueName: \"kubernetes.io/projected/6155ba00-1907-489f-b77e-6b5264b8fac5-kube-api-access-qm6vr\") pod \"6155ba00-1907-489f-b77e-6b5264b8fac5\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") "
Nov 01 09:37:00.053870 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.053799    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-metrics-certs\") pod \"6155ba00-1907-489f-b77e-6b5264b8fac5\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") "
Nov 01 09:37:00.053929 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.053878    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6155ba00-1907-489f-b77e-6b5264b8fac5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6155ba00-1907-489f-b77e-6b5264b8fac5-openshift-service-ca\") pod \"6155ba00-1907-489f-b77e-6b5264b8fac5\" (UID: \"6155ba00-1907-489f-b77e-6b5264b8fac5\") "
Nov 01 09:37:00.054531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.054495    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6155ba00-1907-489f-b77e-6b5264b8fac5-openshift-service-ca" pod "6155ba00-1907-489f-b77e-6b5264b8fac5" (UID: "6155ba00-1907-489f-b77e-6b5264b8fac5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:37:00.057650 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.057612    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6155ba00-1907-489f-b77e-6b5264b8fac5-kube-api-access-qm6vr" pod "6155ba00-1907-489f-b77e-6b5264b8fac5" (UID: "6155ba00-1907-489f-b77e-6b5264b8fac5"). InnerVolumeSpecName "kube-api-access-qm6vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:00.075898 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.075860    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-maas-api-tls" pod "6155ba00-1907-489f-b77e-6b5264b8fac5" (UID: "6155ba00-1907-489f-b77e-6b5264b8fac5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:37:00.126013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.124298    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-metrics-certs" pod "6155ba00-1907-489f-b77e-6b5264b8fac5" (UID: "6155ba00-1907-489f-b77e-6b5264b8fac5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:37:00.156062 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.155980    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6vr\" (UniqueName: \"kubernetes.io/projected/6155ba00-1907-489f-b77e-6b5264b8fac5-kube-api-access-qm6vr\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:00.156062 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.156020    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:00.156062 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.156050    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6155ba00-1907-489f-b77e-6b5264b8fac5-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:00.156062 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.156062    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6155ba00-1907-489f-b77e-6b5264b8fac5-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:00.365821 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.365784    4931 generic.go:356] "Generic (PLEG): container finished" podID="6155ba00-1907-489f-b77e-6b5264b8fac5" containerID="896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a" exitCode=0
Nov 01 09:37:00.366039 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.365822    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs" event={"ID":"6155ba00-1907-489f-b77e-6b5264b8fac5","Type":"ContainerDied","Data":"896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a"}
Nov 01 09:37:00.366039 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.365891    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs" event={"ID":"6155ba00-1907-489f-b77e-6b5264b8fac5","Type":"ContainerDied","Data":"723540b7156c4c6cb64892420626b0f334783a1d34cda6c5457c6b90d7092fd0"}
Nov 01 09:37:00.366039 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.365915    4931 scope.go:122] "RemoveContainer" containerID="896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a"
Nov 01 09:37:00.366039 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.365835    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"
Nov 01 09:37:00.377476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.377455    4931 scope.go:122] "RemoveContainer" containerID="896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a"
Nov 01 09:37:00.377862 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:37:00.377842    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a\": container with ID starting with 896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a not found: ID does not exist" containerID="896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a"
Nov 01 09:37:00.377915 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.377874    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a"} err="failed to get container status \"896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a\": rpc error: code = NotFound desc = could not find container \"896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a\": container with ID starting with 896cabf9f8a079b37bff19d0eee13a02d14d2801ce1caabcf723ef938593618a not found: ID does not exist"
Nov 01 09:37:00.524710 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.524680    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"]
Nov 01 09:37:00.572449 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:00.572415    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25d9fe-ffb4866cf-shvjs"]
Nov 01 09:37:01.023608 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:37:01.023574    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:37:02.047845 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:02.047810    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6155ba00-1907-489f-b77e-6b5264b8fac5" path="/var/lib/kubelet/pods/6155ba00-1907-489f-b77e-6b5264b8fac5/volumes"
Nov 01 09:37:10.743692 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:10.743667    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:37:10.747589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:10.747571    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:37:10.771645 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:10.771624    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:37:10.775342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:10.775320    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:37:20.852228 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:20.852141    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"]
Nov 01 09:37:20.852639 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:20.852453    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds" podUID="f47a72b8-69f2-4033-8160-cd8180fd776a" containerName="maas-api" containerID="cri-o://078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38" gracePeriod=30
Nov 01 09:37:21.169319 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.169292    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:37:21.188963 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.188913    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47a72b8-69f2-4033-8160-cd8180fd776a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47a72b8-69f2-4033-8160-cd8180fd776a-openshift-service-ca\") pod \"f47a72b8-69f2-4033-8160-cd8180fd776a\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") "
Nov 01 09:37:21.189195 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.188998    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47a72b8-69f2-4033-8160-cd8180fd776a-kube-api-access-4n5zm\" (UniqueName: \"kubernetes.io/projected/f47a72b8-69f2-4033-8160-cd8180fd776a-kube-api-access-4n5zm\") pod \"f47a72b8-69f2-4033-8160-cd8180fd776a\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") "
Nov 01 09:37:21.189195 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.189077    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls\") pod \"f47a72b8-69f2-4033-8160-cd8180fd776a\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") "
Nov 01 09:37:21.189331 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.189290    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-metrics-certs\") pod \"f47a72b8-69f2-4033-8160-cd8180fd776a\" (UID: \"f47a72b8-69f2-4033-8160-cd8180fd776a\") "
Nov 01 09:37:21.189331 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.189309    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47a72b8-69f2-4033-8160-cd8180fd776a-openshift-service-ca" pod "f47a72b8-69f2-4033-8160-cd8180fd776a" (UID: "f47a72b8-69f2-4033-8160-cd8180fd776a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:37:21.189726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.189669    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47a72b8-69f2-4033-8160-cd8180fd776a-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:21.191370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.191346    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47a72b8-69f2-4033-8160-cd8180fd776a-kube-api-access-4n5zm" pod "f47a72b8-69f2-4033-8160-cd8180fd776a" (UID: "f47a72b8-69f2-4033-8160-cd8180fd776a"). InnerVolumeSpecName "kube-api-access-4n5zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:21.191935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.191914    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-metrics-certs" pod "f47a72b8-69f2-4033-8160-cd8180fd776a" (UID: "f47a72b8-69f2-4033-8160-cd8180fd776a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:37:21.192044 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.192006    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls" pod "f47a72b8-69f2-4033-8160-cd8180fd776a" (UID: "f47a72b8-69f2-4033-8160-cd8180fd776a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:37:21.290765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.290729    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n5zm\" (UniqueName: \"kubernetes.io/projected/f47a72b8-69f2-4033-8160-cd8180fd776a-kube-api-access-4n5zm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:21.290765 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.290767    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:21.290969 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.290784    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f47a72b8-69f2-4033-8160-cd8180fd776a-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:21.521756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.521666    4931 generic.go:356] "Generic (PLEG): container finished" podID="f47a72b8-69f2-4033-8160-cd8180fd776a" containerID="078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38" exitCode=0
Nov 01 09:37:21.521756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.521720    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"
Nov 01 09:37:21.521992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.521755    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds" event={"ID":"f47a72b8-69f2-4033-8160-cd8180fd776a","Type":"ContainerDied","Data":"078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38"}
Nov 01 09:37:21.521992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.521812    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds" event={"ID":"f47a72b8-69f2-4033-8160-cd8180fd776a","Type":"ContainerDied","Data":"6c0b943d18d6d2f14ca7db823970f0aa917a369bffb320f39aa729a81e9a8de9"}
Nov 01 09:37:21.521992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.521842    4931 scope.go:122] "RemoveContainer" containerID="078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38"
Nov 01 09:37:21.535702 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.535676    4931 scope.go:122] "RemoveContainer" containerID="078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38"
Nov 01 09:37:21.536092 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:37:21.536071    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38\": container with ID starting with 078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38 not found: ID does not exist" containerID="078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38"
Nov 01 09:37:21.536179 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.536108    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38"} err="failed to get container status \"078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38\": rpc error: code = NotFound desc = could not find container \"078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38\": container with ID starting with 078be3e801046b722594796c8678ce0540a7e93ffa096831f37e0d5467fc7e38 not found: ID does not exist"
Nov 01 09:37:21.680749 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.680699    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"]
Nov 01 09:37:21.683706 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:21.683678    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3728d-5bc9c89d49-wnjds"]
Nov 01 09:37:22.052369 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:22.052333    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47a72b8-69f2-4033-8160-cd8180fd776a" path="/var/lib/kubelet/pods/f47a72b8-69f2-4033-8160-cd8180fd776a/volumes"
Nov 01 09:37:31.023341 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:37:31.023305    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:37:46.777788 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:46.777751    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"]
Nov 01 09:37:46.778455 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:46.778038    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz" podUID="064d83c4-5e8f-4aec-9d64-6c4b89453c8d" containerName="maas-api" containerID="cri-o://886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959" gracePeriod=30
Nov 01 09:37:47.093674 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.093644    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:37:47.195351 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.194162    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-openshift-service-ca\") pod \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") "
Nov 01 09:37:47.195351 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.194245    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-metrics-certs\") pod \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") "
Nov 01 09:37:47.195351 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.194301    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-kube-api-access-zrxvp\" (UniqueName: \"kubernetes.io/projected/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-kube-api-access-zrxvp\") pod \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") "
Nov 01 09:37:47.195351 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.194361    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls\") pod \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\" (UID: \"064d83c4-5e8f-4aec-9d64-6c4b89453c8d\") "
Nov 01 09:37:47.195351 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.194563    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-openshift-service-ca" pod "064d83c4-5e8f-4aec-9d64-6c4b89453c8d" (UID: "064d83c4-5e8f-4aec-9d64-6c4b89453c8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:37:47.195351 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.194947    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:47.198402 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.198376    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-kube-api-access-zrxvp" pod "064d83c4-5e8f-4aec-9d64-6c4b89453c8d" (UID: "064d83c4-5e8f-4aec-9d64-6c4b89453c8d"). InnerVolumeSpecName "kube-api-access-zrxvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:37:47.202527 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.202490    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-metrics-certs" pod "064d83c4-5e8f-4aec-9d64-6c4b89453c8d" (UID: "064d83c4-5e8f-4aec-9d64-6c4b89453c8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:37:47.203562 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.203523    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls" pod "064d83c4-5e8f-4aec-9d64-6c4b89453c8d" (UID: "064d83c4-5e8f-4aec-9d64-6c4b89453c8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:37:47.295989 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.295954    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:47.295989 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.295987    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxvp\" (UniqueName: \"kubernetes.io/projected/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-kube-api-access-zrxvp\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:47.295989 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.295997    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064d83c4-5e8f-4aec-9d64-6c4b89453c8d-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:37:47.739046 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.735820    4931 generic.go:356] "Generic (PLEG): container finished" podID="064d83c4-5e8f-4aec-9d64-6c4b89453c8d" containerID="886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959" exitCode=0
Nov 01 09:37:47.739046 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.736088    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz" event={"ID":"064d83c4-5e8f-4aec-9d64-6c4b89453c8d","Type":"ContainerDied","Data":"886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959"}
Nov 01 09:37:47.739046 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.736190    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz" event={"ID":"064d83c4-5e8f-4aec-9d64-6c4b89453c8d","Type":"ContainerDied","Data":"da6a6003fbcdd68858d2c1b978a275887e12bc6ba4e83d9fb71b6817f5e73fd3"}
Nov 01 09:37:47.739046 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.736225    4931 scope.go:122] "RemoveContainer" containerID="886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959"
Nov 01 09:37:47.739046 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.736439    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"
Nov 01 09:37:47.781970 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.781938    4931 scope.go:122] "RemoveContainer" containerID="886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959"
Nov 01 09:37:47.782916 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:37:47.782877    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959\": container with ID starting with 886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959 not found: ID does not exist" containerID="886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959"
Nov 01 09:37:47.783044 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.782925    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959"} err="failed to get container status \"886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959\": rpc error: code = NotFound desc = could not find container \"886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959\": container with ID starting with 886f2c4ca2a077dafef99db6b8e12f58eed93c9b441116bbc472d2cd2c946959 not found: ID does not exist"
Nov 01 09:37:47.898257 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.898219    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"]
Nov 01 09:37:47.945866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:47.945832    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-49e170-84c8f48474-bhbqz"]
Nov 01 09:37:48.048178 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:37:48.048094    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064d83c4-5e8f-4aec-9d64-6c4b89453c8d" path="/var/lib/kubelet/pods/064d83c4-5e8f-4aec-9d64-6c4b89453c8d/volumes"
Nov 01 09:38:01.024070 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:38:01.024011    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:38:13.224349 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.224317    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"]
Nov 01 09:38:13.224844 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.224646    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz" podUID="3defcc7b-114e-4175-98b5-e7ae11495867" containerName="maas-api" containerID="cri-o://5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0" gracePeriod=30
Nov 01 09:38:13.511881 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.511855    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:38:13.616729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.615488    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3defcc7b-114e-4175-98b5-e7ae11495867-kube-api-access-5g5bt\" (UniqueName: \"kubernetes.io/projected/3defcc7b-114e-4175-98b5-e7ae11495867-kube-api-access-5g5bt\") pod \"3defcc7b-114e-4175-98b5-e7ae11495867\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") "
Nov 01 09:38:13.616729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.615578    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-metrics-certs\") pod \"3defcc7b-114e-4175-98b5-e7ae11495867\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") "
Nov 01 09:38:13.616729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.615662    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3defcc7b-114e-4175-98b5-e7ae11495867-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3defcc7b-114e-4175-98b5-e7ae11495867-openshift-service-ca\") pod \"3defcc7b-114e-4175-98b5-e7ae11495867\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") "
Nov 01 09:38:13.616729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.615745    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls\") pod \"3defcc7b-114e-4175-98b5-e7ae11495867\" (UID: \"3defcc7b-114e-4175-98b5-e7ae11495867\") "
Nov 01 09:38:13.617821 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.617774    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3defcc7b-114e-4175-98b5-e7ae11495867-openshift-service-ca" pod "3defcc7b-114e-4175-98b5-e7ae11495867" (UID: "3defcc7b-114e-4175-98b5-e7ae11495867"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:38:13.635095 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.629452    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls" pod "3defcc7b-114e-4175-98b5-e7ae11495867" (UID: "3defcc7b-114e-4175-98b5-e7ae11495867"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:38:13.636368 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.636319    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-metrics-certs" pod "3defcc7b-114e-4175-98b5-e7ae11495867" (UID: "3defcc7b-114e-4175-98b5-e7ae11495867"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:38:13.656038 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.655983    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3defcc7b-114e-4175-98b5-e7ae11495867-kube-api-access-5g5bt" pod "3defcc7b-114e-4175-98b5-e7ae11495867" (UID: "3defcc7b-114e-4175-98b5-e7ae11495867"). InnerVolumeSpecName "kube-api-access-5g5bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:38:13.717261 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.717229    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:13.717261 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.717259    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5bt\" (UniqueName: \"kubernetes.io/projected/3defcc7b-114e-4175-98b5-e7ae11495867-kube-api-access-5g5bt\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:13.717482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.717271    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3defcc7b-114e-4175-98b5-e7ae11495867-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:13.717482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.717281    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3defcc7b-114e-4175-98b5-e7ae11495867-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:38:13.911426 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.911393    4931 generic.go:356] "Generic (PLEG): container finished" podID="3defcc7b-114e-4175-98b5-e7ae11495867" containerID="5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0" exitCode=0
Nov 01 09:38:13.911618 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.911440    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"
Nov 01 09:38:13.911618 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.911486    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz" event={"ID":"3defcc7b-114e-4175-98b5-e7ae11495867","Type":"ContainerDied","Data":"5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0"}
Nov 01 09:38:13.911618 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.911545    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz" event={"ID":"3defcc7b-114e-4175-98b5-e7ae11495867","Type":"ContainerDied","Data":"73dad07b8681e25b08482eb91d60ae796844c1919290298211a5396e83492bd9"}
Nov 01 09:38:13.911618 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.911574    4931 scope.go:122] "RemoveContainer" containerID="5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0"
Nov 01 09:38:13.922905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.922881    4931 scope.go:122] "RemoveContainer" containerID="5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0"
Nov 01 09:38:13.923215 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:38:13.923198    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0\": container with ID starting with 5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0 not found: ID does not exist" containerID="5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0"
Nov 01 09:38:13.923301 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:13.923226    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0"} err="failed to get container status \"5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0\": rpc error: code = NotFound desc = could not find container \"5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0\": container with ID starting with 5a7b8127864ce9c4ba458b8797b01ec88baf679c278bd394eeeab20cbd76d4f0 not found: ID does not exist"
Nov 01 09:38:14.070943 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:14.070906    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"]
Nov 01 09:38:14.076116 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:14.076090    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f3121-5f4f85d75b-7dzhz"]
Nov 01 09:38:16.047933 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:16.047898    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3defcc7b-114e-4175-98b5-e7ae11495867" path="/var/lib/kubelet/pods/3defcc7b-114e-4175-98b5-e7ae11495867/volumes"
Nov 01 09:38:28.643779 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.643738    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"]
Nov 01 09:38:28.644323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644271    4931 cpu_manager.go:414] "containerMap: removing container" podUID="064d83c4-5e8f-4aec-9d64-6c4b89453c8d" containerName="maas-api"
Nov 01 09:38:28.644323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644284    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064d83c4-5e8f-4aec-9d64-6c4b89453c8d" containerName="maas-api"
Nov 01 09:38:28.644323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644297    4931 cpu_manager.go:414] "containerMap: removing container" podUID="f47a72b8-69f2-4033-8160-cd8180fd776a" containerName="maas-api"
Nov 01 09:38:28.644323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644301    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47a72b8-69f2-4033-8160-cd8180fd776a" containerName="maas-api"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644326    4931 cpu_manager.go:414] "containerMap: removing container" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644331    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644346    4931 cpu_manager.go:414] "containerMap: removing container" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644350    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644357    4931 cpu_manager.go:414] "containerMap: removing container" podUID="3defcc7b-114e-4175-98b5-e7ae11495867" containerName="maas-api"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644361    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3defcc7b-114e-4175-98b5-e7ae11495867" containerName="maas-api"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644372    4931 cpu_manager.go:414] "containerMap: removing container" podUID="6155ba00-1907-489f-b77e-6b5264b8fac5" containerName="maas-api"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644376    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6155ba00-1907-489f-b77e-6b5264b8fac5" containerName="maas-api"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644449    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="064d83c4-5e8f-4aec-9d64-6c4b89453c8d" containerName="maas-api"
Nov 01 09:38:28.644459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644457    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47a72b8-69f2-4033-8160-cd8180fd776a" containerName="maas-api"
Nov 01 09:38:28.644756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644468    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="872aebee-a5d5-4a7e-a884-7cf86ab02777" containerName="istio-proxy"
Nov 01 09:38:28.644756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644476    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee58f10-6561-41fe-bcc2-d25018b3ce66" containerName="istio-proxy"
Nov 01 09:38:28.644756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644481    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3defcc7b-114e-4175-98b5-e7ae11495867" containerName="maas-api"
Nov 01 09:38:28.644756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.644487    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6155ba00-1907-489f-b77e-6b5264b8fac5" containerName="maas-api"
Nov 01 09:38:28.648397 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.648374    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696055 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.695976    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.696113    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696335 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.696245    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696335 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.696306    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696444 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.696362    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696444 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.696435    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696603 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.696583    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.696618    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dzj\" (UniqueName: \"kubernetes.io/projected/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-kube-api-access-l8dzj\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.696664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.696657    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.701911 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.701880    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kpfx2\""
Nov 01 09:38:28.752817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.752778    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"]
Nov 01 09:38:28.797545 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797505    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.797756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797561    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.797756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797616    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.797756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797638    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.797756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797654    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.797756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797689    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.797756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797743    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.798126 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797759    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dzj\" (UniqueName: \"kubernetes.io/projected/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-kube-api-access-l8dzj\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.798126 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.797785    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.798126 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.798065    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.798771 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.798614    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.798771 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.798689    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.798771 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.798691    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.799074 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.799050    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.800414 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.800387    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.801827 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.801805    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.857724 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.857687    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.858057 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.858007    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dzj\" (UniqueName: \"kubernetes.io/projected/ca5beb2f-1ccf-40e5-a862-e0264dbd35e4-kube-api-access-l8dzj\") pod \"maas-default-gateway-openshift-default-746ff8f49f-5gxgq\" (UID: \"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:28.962878 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:28.962785    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:29.408919 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:29.408882    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"]
Nov 01 09:38:29.410650 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:38:29.410622    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5beb2f_1ccf_40e5_a862_e0264dbd35e4.slice/crio-cd8c6a1bc8d78968ba70f6b58f2c721cbfef2330a57fbfc98367374f7c60e53f WatchSource:0}: Error finding container cd8c6a1bc8d78968ba70f6b58f2c721cbfef2330a57fbfc98367374f7c60e53f: Status 404 returned error can't find the container with id cd8c6a1bc8d78968ba70f6b58f2c721cbfef2330a57fbfc98367374f7c60e53f
Nov 01 09:38:29.413342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:29.413302    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:38:29.413435 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:29.413389    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:38:29.413483 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:29.413431    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:38:30.026694 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:30.026659    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" event={"ID":"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4","Type":"ContainerStarted","Data":"313339ea09ea5dfa20be8d94b42a5e33516cbd220f8d63661a1655cc391e86a0"}
Nov 01 09:38:30.027273 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:30.026714    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" event={"ID":"ca5beb2f-1ccf-40e5-a862-e0264dbd35e4","Type":"ContainerStarted","Data":"cd8c6a1bc8d78968ba70f6b58f2c721cbfef2330a57fbfc98367374f7c60e53f"}
Nov 01 09:38:30.144496 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:30.144446    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podStartSLOduration=2.144433978 podStartE2EDuration="2.144433978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:30.144183643 +0000 UTC m=+3980.259056612" watchObservedRunningTime="2026-11-01 09:38:30.144433978 +0000 UTC m=+3980.259306946"
Nov 01 09:38:30.974330 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:30.974296    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:30.975288 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:30.975260    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 01 09:38:30.975497 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:30.975472    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 01 09:38:31.028463 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:38:31.028430    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:38:31.966927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:31.966450    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 01 09:38:31.966927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:31.966517    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 01 09:38:32.967513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:32.967475    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 01 09:38:32.968164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:32.968135    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 01 09:38:33.977517 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:33.977478    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 01 09:38:33.978254 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:33.978222    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 01 09:38:34.966060 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:34.964385    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 01 09:38:34.966060 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:34.964447    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 01 09:38:35.964004 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:35.963962    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 01 09:38:35.964513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:35.964038    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 01 09:38:36.963672 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:36.963630    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 01 09:38:36.963874 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:36.963720    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 01 09:38:37.964338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:37.964291    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 01 09:38:37.964737 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:37.964377    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 01 09:38:39.167729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:39.167703    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:39.168240 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:39.168222    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:39.168917 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:39.168888    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq"
Nov 01 09:38:43.775657 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.775618    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"]
Nov 01 09:38:43.780394 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.780373    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.882330 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.882294    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"]
Nov 01 09:38:43.920118 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920079    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.920305 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920130    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhj9\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-kube-api-access-rmhj9\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.920305 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920201    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.920305 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920243    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.920412 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920321    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.920412 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920357    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.920412 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920383    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.920507 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920446    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:43.920507 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:43.920469    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e640a6-6a90-4f20-aecb-8470965c2f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.021550 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021511    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.021782 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021587    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.021782 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021625    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.021782 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021660    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.021782 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021708    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.021782 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021732    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e640a6-6a90-4f20-aecb-8470965c2f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.022098 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021865    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.022098 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021904    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhj9\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-kube-api-access-rmhj9\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.022098 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.021940    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.022098 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.022065    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.022327 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.022299    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.022389 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.022341    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.022448 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.022426    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.022648 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.022628    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e640a6-6a90-4f20-aecb-8470965c2f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.024265 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.024244    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.024476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.024459    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.081482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.081400    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.081650 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.081595    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhj9\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-kube-api-access-rmhj9\") pod \"maas-default-gateway-openshift-default-746ff8f49f-4m4gs\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.094455 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.094421    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:44.519716 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.519687    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"]
Nov 01 09:38:44.520643 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:38:44.520614    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e640a6_6a90_4f20_aecb_8470965c2f0c.slice/crio-892789fd3114b37c906e97d5ccb8aec74cf3df872688a743f7f63dd6910446a3 WatchSource:0}: Error finding container 892789fd3114b37c906e97d5ccb8aec74cf3df872688a743f7f63dd6910446a3: Status 404 returned error can't find the container with id 892789fd3114b37c906e97d5ccb8aec74cf3df872688a743f7f63dd6910446a3
Nov 01 09:38:44.523870 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.523839    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:38:44.523952 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.523902    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:38:44.523991 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:44.523950    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:38:45.138099 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:45.138060    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" event={"ID":"40e640a6-6a90-4f20-aecb-8470965c2f0c","Type":"ContainerStarted","Data":"1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f"}
Nov 01 09:38:45.138615 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:45.138138    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" event={"ID":"40e640a6-6a90-4f20-aecb-8470965c2f0c","Type":"ContainerStarted","Data":"892789fd3114b37c906e97d5ccb8aec74cf3df872688a743f7f63dd6910446a3"}
Nov 01 09:38:45.256782 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:45.256729    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podStartSLOduration=2.256716195 podStartE2EDuration="2.256716195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:45.255322597 +0000 UTC m=+3995.370195568" watchObservedRunningTime="2026-11-01 09:38:45.256716195 +0000 UTC m=+3995.371589163"
Nov 01 09:38:46.095407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:46.095369    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:46.097191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:46.097163    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 01 09:38:46.097339 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:46.097219    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 01 09:38:47.095214 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:47.095175    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 01 09:38:47.095631 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:47.095231    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 01 09:38:48.095636 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:48.095599    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 01 09:38:48.096091 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:48.095652    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 01 09:38:49.095164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:49.095129    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 01 09:38:49.095358 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:49.095185    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 01 09:38:50.095591 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:50.095558    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 01 09:38:50.096004 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:50.095611    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 01 09:38:51.095572 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:51.095534    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 01 09:38:51.095786 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:51.095593    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 01 09:38:52.095164 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:52.095117    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 01 09:38:52.095395 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:52.095189    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 01 09:38:54.096294 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:54.096248    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:38:54.096833 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:54.096320    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:38:54.121748 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:54.121702    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:54.122310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:54.122284    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:38:54.123541 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:38:54.123520    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 09:39:01.024199 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:39:01.024148    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:39:16.766459 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:16.766420    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"]
Nov 01 09:39:16.766981 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:16.766668    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6" podUID="cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" containerName="maas-api" containerID="cri-o://9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a" gracePeriod=30
Nov 01 09:39:17.065380 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.065355    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:39:17.152815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.152782    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-metrics-certs\") pod \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") "
Nov 01 09:39:17.153058 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.152854    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls\") pod \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") "
Nov 01 09:39:17.153058 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.153017    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-kube-api-access-flhcq\" (UniqueName: \"kubernetes.io/projected/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-kube-api-access-flhcq\") pod \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") "
Nov 01 09:39:17.153255 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.153235    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-openshift-service-ca\") pod \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\" (UID: \"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737\") "
Nov 01 09:39:17.153672 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.153641    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-openshift-service-ca" pod "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" (UID: "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:39:17.153955 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.153915    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:17.155781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.155747    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-metrics-certs" pod "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" (UID: "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:39:17.155877 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.155779    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls" pod "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" (UID: "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:39:17.155922 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.155900    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-kube-api-access-flhcq" pod "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" (UID: "cdf4d3e3-c5c6-4c54-8b2e-6d693c144737"). InnerVolumeSpecName "kube-api-access-flhcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:39:17.255425 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.255392    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:17.255425 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.255422    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:17.255425 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.255434    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flhcq\" (UniqueName: \"kubernetes.io/projected/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737-kube-api-access-flhcq\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:39:17.371781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.371741    4931 generic.go:356] "Generic (PLEG): container finished" podID="cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" containerID="9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a" exitCode=0
Nov 01 09:39:17.371956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.371792    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6" event={"ID":"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737","Type":"ContainerDied","Data":"9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a"}
Nov 01 09:39:17.371956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.371798    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"
Nov 01 09:39:17.371956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.371841    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6" event={"ID":"cdf4d3e3-c5c6-4c54-8b2e-6d693c144737","Type":"ContainerDied","Data":"a8a058414a3e2790d11428b0406ce5d8c711f1ee3be5301b885917aad2c9ecd1"}
Nov 01 09:39:17.371956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.371864    4931 scope.go:122] "RemoveContainer" containerID="9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a"
Nov 01 09:39:17.382767 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.382750    4931 scope.go:122] "RemoveContainer" containerID="9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a"
Nov 01 09:39:17.383094 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:39:17.383073    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a\": container with ID starting with 9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a not found: ID does not exist" containerID="9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a"
Nov 01 09:39:17.383181 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.383104    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a"} err="failed to get container status \"9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a\": rpc error: code = NotFound desc = could not find container \"9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a\": container with ID starting with 9e8d50a5504e8f33e00190b1811adb8f197ea8497d14452c6e9362f7442f855a not found: ID does not exist"
Nov 01 09:39:17.531817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.531779    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"]
Nov 01 09:39:17.535935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:17.535912    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b0d0f3-7dff99799d-vv4w6"]
Nov 01 09:39:18.048556 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:18.048521    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" path="/var/lib/kubelet/pods/cdf4d3e3-c5c6-4c54-8b2e-6d693c144737/volumes"
Nov 01 09:39:31.024349 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:39:31.024310    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:39:38.845926 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:38.845893    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"]
Nov 01 09:39:38.846444 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:38.846429    4931 cpu_manager.go:414] "containerMap: removing container" podUID="cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" containerName="maas-api"
Nov 01 09:39:38.846493 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:38.846445    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" containerName="maas-api"
Nov 01 09:39:38.846554 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:38.846543    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf4d3e3-c5c6-4c54-8b2e-6d693c144737" containerName="maas-api"
Nov 01 09:39:38.849794 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:38.849777    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:38.902191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:38.902158    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-cddc5b-serving-cert\""
Nov 01 09:39:38.953118 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:38.953085    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"]
Nov 01 09:39:39.010573 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.010524    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvx8\" (UniqueName: \"kubernetes.io/projected/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-kube-api-access-skvx8\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.010753 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.010631    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.010753 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.010717    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.010900 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.010880    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-metrics-certs\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.112395 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.112368    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-metrics-certs\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.112565 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.112447    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skvx8\" (UniqueName: \"kubernetes.io/projected/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-kube-api-access-skvx8\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.112565 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.112493    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.112565 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.112516    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.113280 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.113254    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.114985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.114966    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-metrics-certs\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.115049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.115008    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.172661 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.172629    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvx8\" (UniqueName: \"kubernetes.io/projected/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-kube-api-access-skvx8\") pod \"maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.463484 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.463392    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:39.848811 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:39.848780    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"]
Nov 01 09:39:39.850158 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:39:39.850132    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4753eca6_0326_4eaa_b4ca_63e2a5caafb4.slice/crio-44d6eaad5a3b5fe7f6798ec4e9746f86fefca29693e2d846b86a991d6bb71419 WatchSource:0}: Error finding container 44d6eaad5a3b5fe7f6798ec4e9746f86fefca29693e2d846b86a991d6bb71419: Status 404 returned error can't find the container with id 44d6eaad5a3b5fe7f6798ec4e9746f86fefca29693e2d846b86a991d6bb71419
Nov 01 09:39:40.564516 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:40.564480    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj" event={"ID":"4753eca6-0326-4eaa-b4ca-63e2a5caafb4","Type":"ContainerStarted","Data":"44d6eaad5a3b5fe7f6798ec4e9746f86fefca29693e2d846b86a991d6bb71419"}
Nov 01 09:39:41.573603 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:41.573564    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj" event={"ID":"4753eca6-0326-4eaa-b4ca-63e2a5caafb4","Type":"ContainerStarted","Data":"5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa"}
Nov 01 09:39:41.574106 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:41.573656    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:41.688870 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:41.688815    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj" podStartSLOduration=2.742817637 podStartE2EDuration="3.6888s" totalImagesPullingTime="945.982363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:39:41.686855513 +0000 UTC m=+4051.801728484" watchObservedRunningTime="2026-11-01 09:39:41.6888 +0000 UTC m=+4051.803672967"
Nov 01 09:39:47.592052 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:47.591995    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:39:53.354273 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.354234    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"]
Nov 01 09:39:53.358239 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.358222    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.409959 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.409931    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3a53f2-serving-cert\""
Nov 01 09:39:53.460998 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.460965    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"]
Nov 01 09:39:53.501636 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.501607    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-metrics-certs\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.501816 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.501751    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d9f2ae-4845-4ba9-82b1-9f926b054e35-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.501869 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.501834    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.501957 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.501932    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gct8z\" (UniqueName: \"kubernetes.io/projected/93d9f2ae-4845-4ba9-82b1-9f926b054e35-kube-api-access-gct8z\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.605980 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.605877    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.605980 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.605925    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gct8z\" (UniqueName: \"kubernetes.io/projected/93d9f2ae-4845-4ba9-82b1-9f926b054e35-kube-api-access-gct8z\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.606274 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.606053    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-metrics-certs\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.606274 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.606149    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d9f2ae-4845-4ba9-82b1-9f926b054e35-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.606589 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:39:53.606554    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-serving-cert: secret "maas-api-e2e-inf-a-3a53f2-serving-cert" not found
Nov 01 09:39:53.606743 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:39:53.606638    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls podName:93d9f2ae-4845-4ba9-82b1-9f926b054e35 nodeName:}" failed. No retries permitted until 2026-11-01 09:39:54.106616866 +0000 UTC m=+4064.221489811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls") pod "maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc" (UID: "93d9f2ae-4845-4ba9-82b1-9f926b054e35") : secret "maas-api-e2e-inf-a-3a53f2-serving-cert" not found
Nov 01 09:39:53.606958 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.606932    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d9f2ae-4845-4ba9-82b1-9f926b054e35-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.609319 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.609297    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-metrics-certs\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:53.669897 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:53.669865    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gct8z\" (UniqueName: \"kubernetes.io/projected/93d9f2ae-4845-4ba9-82b1-9f926b054e35-kube-api-access-gct8z\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:54.112802 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:54.112763    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:54.115626 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:54.115597    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:54.272778 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:54.272739    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:54.654191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:54.654161    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"]
Nov 01 09:39:54.667718 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:54.667690    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc" event={"ID":"93d9f2ae-4845-4ba9-82b1-9f926b054e35","Type":"ContainerStarted","Data":"928f185c55d036d3d990ea119d5bd30bec322b66470a27ea671cd62edb1341ed"}
Nov 01 09:39:55.677927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:55.677883    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc" event={"ID":"93d9f2ae-4845-4ba9-82b1-9f926b054e35","Type":"ContainerStarted","Data":"254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c"}
Nov 01 09:39:55.678431 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:55.677986    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:39:55.793982 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:55.793882    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc" podStartSLOduration=1.975420963 podStartE2EDuration="2.793867551s" totalImagesPullingTime="818.446588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:39:55.791595648 +0000 UTC m=+4065.906468618" watchObservedRunningTime="2026-11-01 09:39:55.793867551 +0000 UTC m=+4065.908740521"
Nov 01 09:39:59.369095 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.369058    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"]
Nov 01 09:39:59.373235 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.373217    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477147 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477108    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477171    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477255    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477306    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477383    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477400    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a1f4a0-3751-45de-9fc7-89188a792f50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477536    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477606    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.477985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.477635    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljs2z\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-kube-api-access-ljs2z\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.579398 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579366    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"]
Nov 01 09:39:59.579589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579396    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.579589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579478    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.579589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579517    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.579589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579562    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.579823 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579637    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.579823 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579665    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a1f4a0-3751-45de-9fc7-89188a792f50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.579823 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579772    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.579823 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579781    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.580131 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579840    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.580131 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.579869    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljs2z\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-kube-api-access-ljs2z\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.580789 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.580763    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a1f4a0-3751-45de-9fc7-89188a792f50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.581067 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.581043    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.581318 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.581297    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.581506 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.581280    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.582842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.582822    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.583745 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.583724    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.638567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.638479    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.638774 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.638749    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljs2z\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-kube-api-access-ljs2z\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rvlsf\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:39:59.686034 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:39:59.685983    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:40:00.117630 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:00.117596    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"]
Nov 01 09:40:00.118010 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:40:00.117987    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a1f4a0_3751_45de_9fc7_89188a792f50.slice/crio-7d89a3f41de40e1040442f97009bd6a053d308eb3c0ffa27a90fa3b30b8372bb WatchSource:0}: Error finding container 7d89a3f41de40e1040442f97009bd6a053d308eb3c0ffa27a90fa3b30b8372bb: Status 404 returned error can't find the container with id 7d89a3f41de40e1040442f97009bd6a053d308eb3c0ffa27a90fa3b30b8372bb
Nov 01 09:40:00.120702 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:00.120665    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:40:00.120768 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:00.120742    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:40:00.120805 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:00.120772    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:40:00.713842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:00.713804    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" event={"ID":"a8a1f4a0-3751-45de-9fc7-89188a792f50","Type":"ContainerStarted","Data":"55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f"}
Nov 01 09:40:00.714262 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:00.713870    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" event={"ID":"a8a1f4a0-3751-45de-9fc7-89188a792f50","Type":"ContainerStarted","Data":"7d89a3f41de40e1040442f97009bd6a053d308eb3c0ffa27a90fa3b30b8372bb"}
Nov 01 09:40:00.933729 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:00.933676    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podStartSLOduration=1.933661656 podStartE2EDuration="1.933661656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:40:00.885540968 +0000 UTC m=+4071.000413937" watchObservedRunningTime="2026-11-01 09:40:00.933661656 +0000 UTC m=+4071.048534624"
Nov 01 09:40:01.022532 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:40:01.022442    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:40:01.686477 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:01.686433    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:40:01.688260 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:01.688217    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 09:40:01.688410 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:01.688274    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 09:40:01.691142 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:01.691120    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:40:02.686590 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:02.686550    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 09:40:02.687069 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:02.686635    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 09:40:03.688560 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:03.688520    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 09:40:03.689065 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:03.688608    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 09:40:04.687290 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:04.687243    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 09:40:04.687597 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:04.687563    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 09:40:05.686538 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:05.686495    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 09:40:05.687071 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:05.686551    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 09:40:06.686667 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:06.686628    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 09:40:06.687238 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:06.686729    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 09:40:07.687143 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:07.687100    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 09:40:07.687691 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:07.687184    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 09:40:09.687476 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:09.687437    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:40:09.687992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:09.687531    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:40:09.753803 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:09.753762    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:40:09.754390 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:09.754361    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:40:09.755370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:09.755350    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:40:27.224074 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.223970    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"]
Nov 01 09:40:27.228263 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.228241    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.244443 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.244409    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.244619 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.244529    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.244619 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.244582    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-metrics-certs\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.244813 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.244793    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpsf\" (UniqueName: \"kubernetes.io/projected/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-kube-api-access-9bpsf\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.279658 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.279618    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-12a01a-serving-cert\""
Nov 01 09:40:27.330389 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.330356    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"]
Nov 01 09:40:27.346364 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.346330    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpsf\" (UniqueName: \"kubernetes.io/projected/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-kube-api-access-9bpsf\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.346531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.346377    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.346531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.346425    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.346531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.346490    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-metrics-certs\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.346639 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:40:27.346530    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-serving-cert: secret "maas-api-e2e-inf-a-12a01a-serving-cert" not found
Nov 01 09:40:27.346639 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:40:27.346593    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls podName:def75dd7-bd4b-4b93-a463-a4fbbbeb85a2 nodeName:}" failed. No retries permitted until 2026-11-01 09:40:27.846572122 +0000 UTC m=+4097.961445069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls") pod "maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x" (UID: "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2") : secret "maas-api-e2e-inf-a-12a01a-serving-cert" not found
Nov 01 09:40:27.347167 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.347150    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.349075 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.349052    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-metrics-certs\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.405572 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.405543    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpsf\" (UniqueName: \"kubernetes.io/projected/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-kube-api-access-9bpsf\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.852094 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.852062    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:27.854679 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:27.854660    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:28.142253 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:28.142213    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:28.515305 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:28.515275    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"]
Nov 01 09:40:28.516067 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:40:28.516018    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef75dd7_bd4b_4b93_a463_a4fbbbeb85a2.slice/crio-543ae32af27ae36a57fc9bd3a4206b2d0f31dab0773b2cb4c82731df8d0fcdd6 WatchSource:0}: Error finding container 543ae32af27ae36a57fc9bd3a4206b2d0f31dab0773b2cb4c82731df8d0fcdd6: Status 404 returned error can't find the container with id 543ae32af27ae36a57fc9bd3a4206b2d0f31dab0773b2cb4c82731df8d0fcdd6
Nov 01 09:40:28.517592 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:28.517571    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:40:28.915773 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:28.915735    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x" event={"ID":"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2","Type":"ContainerStarted","Data":"543ae32af27ae36a57fc9bd3a4206b2d0f31dab0773b2cb4c82731df8d0fcdd6"}
Nov 01 09:40:29.926295 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:29.926185    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x" event={"ID":"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2","Type":"ContainerStarted","Data":"e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7"}
Nov 01 09:40:29.926723 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:29.926412    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:30.041119 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:30.041054    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x" podStartSLOduration=1.918855541 podStartE2EDuration="3.041039047s" totalImagesPullingTime="1.122183506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:40:30.038913407 +0000 UTC m=+4100.153786375" watchObservedRunningTime="2026-11-01 09:40:30.041039047 +0000 UTC m=+4100.155912009"
Nov 01 09:40:31.024380 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:40:31.024337    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:40:35.939229 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:35.939194    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:40:51.230281 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.230243    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"]
Nov 01 09:40:51.234327 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.234307    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.285855 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.285821    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-49f18f-serving-cert\""
Nov 01 09:40:51.336670 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.336638    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"]
Nov 01 09:40:51.340097 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.340075    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.340234 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.340205    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f598d227-7b48-4f40-aa74-c0421b67266e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.340275 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.340253    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtlw\" (UniqueName: \"kubernetes.io/projected/f598d227-7b48-4f40-aa74-c0421b67266e-kube-api-access-vrtlw\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.340354 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.340341    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-metrics-certs\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.441086 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.441044    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.441266 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.441113    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f598d227-7b48-4f40-aa74-c0421b67266e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.441266 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:40:51.441132    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-serving-cert: secret "maas-api-e2e-inf-a-49f18f-serving-cert" not found
Nov 01 09:40:51.441266 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.441135    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtlw\" (UniqueName: \"kubernetes.io/projected/f598d227-7b48-4f40-aa74-c0421b67266e-kube-api-access-vrtlw\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.441266 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:40:51.441202    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls podName:f598d227-7b48-4f40-aa74-c0421b67266e nodeName:}" failed. No retries permitted until 2026-11-01 09:40:51.941180359 +0000 UTC m=+4122.056053317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls") pod "maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt" (UID: "f598d227-7b48-4f40-aa74-c0421b67266e") : secret "maas-api-e2e-inf-a-49f18f-serving-cert" not found
Nov 01 09:40:51.441426 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.441313    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-metrics-certs\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.441789 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.441769    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f598d227-7b48-4f40-aa74-c0421b67266e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.444189 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.444166    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-metrics-certs\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.509855 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.509765    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtlw\" (UniqueName: \"kubernetes.io/projected/f598d227-7b48-4f40-aa74-c0421b67266e-kube-api-access-vrtlw\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.947534 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.947499    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:51.950177 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:51.950146    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls\") pod \"maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:52.150419 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:52.150381    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:52.524853 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:52.524822    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"]
Nov 01 09:40:52.526300 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:40:52.526272    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf598d227_7b48_4f40_aa74_c0421b67266e.slice/crio-ddd9c58a25dcec0477546b2cb56496efc250053763b4ffb8a949f6ee763418df WatchSource:0}: Error finding container ddd9c58a25dcec0477546b2cb56496efc250053763b4ffb8a949f6ee763418df: Status 404 returned error can't find the container with id ddd9c58a25dcec0477546b2cb56496efc250053763b4ffb8a949f6ee763418df
Nov 01 09:40:53.098831 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:53.098787    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt" event={"ID":"f598d227-7b48-4f40-aa74-c0421b67266e","Type":"ContainerStarted","Data":"ddd9c58a25dcec0477546b2cb56496efc250053763b4ffb8a949f6ee763418df"}
Nov 01 09:40:54.108094 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:54.108056    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt" event={"ID":"f598d227-7b48-4f40-aa74-c0421b67266e","Type":"ContainerStarted","Data":"3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009"}
Nov 01 09:40:54.108605 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:54.108138    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:40:54.222687 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:40:54.222632    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt" podStartSLOduration=2.344172985 podStartE2EDuration="3.222617352s" totalImagesPullingTime="878.444367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:40:54.22068483 +0000 UTC m=+4124.335557822" watchObservedRunningTime="2026-11-01 09:40:54.222617352 +0000 UTC m=+4124.337490317"
Nov 01 09:41:00.121285 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:41:00.121254    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:41:01.023493 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:41:01.023438    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:41:31.022117 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:41:31.022078    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:42:01.026097 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:42:01.025982    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:42:10.792795 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:10.792668    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:42:10.798089 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:10.796916    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:42:10.818635 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:10.818610    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:42:10.821971 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:10.821951    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:42:31.024584 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:42:31.024547    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:42:51.546166 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.546131    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"]
Nov 01 09:42:51.550842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.550819    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.600896 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.600867    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c81a9d-serving-cert\""
Nov 01 09:42:51.603960 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.603927    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"]
Nov 01 09:42:51.666490 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.666457    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwrc\" (UniqueName: \"kubernetes.io/projected/310768c0-2031-445b-8014-d7c4ecef7fb4-kube-api-access-4zwrc\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.666684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.666536    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310768c0-2031-445b-8014-d7c4ecef7fb4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.666684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.666648    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-metrics-certs\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.666807 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.666788    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.768491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.768450    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwrc\" (UniqueName: \"kubernetes.io/projected/310768c0-2031-445b-8014-d7c4ecef7fb4-kube-api-access-4zwrc\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.768671 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.768529    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310768c0-2031-445b-8014-d7c4ecef7fb4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.768671 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.768565    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-metrics-certs\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.768671 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.768644    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.769270 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.769241    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310768c0-2031-445b-8014-d7c4ecef7fb4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.771240 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.771214    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-metrics-certs\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.771452 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.771264    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.829610 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.829537    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwrc\" (UniqueName: \"kubernetes.io/projected/310768c0-2031-445b-8014-d7c4ecef7fb4-kube-api-access-4zwrc\") pod \"maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:51.865048 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:51.864995    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:52.244809 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:42:52.244775    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310768c0_2031_445b_8014_d7c4ecef7fb4.slice/crio-47650dbdb3589f93afc26d8cf32308911dc76a7f198c9a2d7c69fe534369f7c7 WatchSource:0}: Error finding container 47650dbdb3589f93afc26d8cf32308911dc76a7f198c9a2d7c69fe534369f7c7: Status 404 returned error can't find the container with id 47650dbdb3589f93afc26d8cf32308911dc76a7f198c9a2d7c69fe534369f7c7
Nov 01 09:42:52.245192 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:52.245170    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"]
Nov 01 09:42:52.956014 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:52.955974    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc" event={"ID":"310768c0-2031-445b-8014-d7c4ecef7fb4","Type":"ContainerStarted","Data":"47650dbdb3589f93afc26d8cf32308911dc76a7f198c9a2d7c69fe534369f7c7"}
Nov 01 09:42:53.964236 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:53.964196    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc" event={"ID":"310768c0-2031-445b-8014-d7c4ecef7fb4","Type":"ContainerStarted","Data":"30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78"}
Nov 01 09:42:53.964650 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:53.964395    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:42:54.078899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:54.078846    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc" podStartSLOduration=2.157801631 podStartE2EDuration="3.07883189s" totalImagesPullingTime="921.030259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:42:54.076776179 +0000 UTC m=+4244.191649147" watchObservedRunningTime="2026-11-01 09:42:54.07883189 +0000 UTC m=+4244.193704857"
Nov 01 09:42:59.975274 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:42:59.975246    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:43:01.029182 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:43:01.029143    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:43:10.974316 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:10.974278    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"]
Nov 01 09:43:10.999405 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:10.999356    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.032906 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.032869    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"]
Nov 01 09:43:11.076287 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.076248    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b3d6bb-serving-cert\""
Nov 01 09:43:11.093203 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.093171    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwsp\" (UniqueName: \"kubernetes.io/projected/55b1b828-311a-4e06-891f-0daeb0fbadd1-kube-api-access-gwwsp\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.093203 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.093208    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-metrics-certs\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.093464 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.093346    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.093464 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.093373    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b1b828-311a-4e06-891f-0daeb0fbadd1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.194229 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.194193    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwsp\" (UniqueName: \"kubernetes.io/projected/55b1b828-311a-4e06-891f-0daeb0fbadd1-kube-api-access-gwwsp\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.194229 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.194231    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-metrics-certs\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.194545 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.194314    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.194545 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:43:11.194401    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-serving-cert: secret "maas-api-e2e-inf-b-b3d6bb-serving-cert" not found
Nov 01 09:43:11.194545 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.194439    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b1b828-311a-4e06-891f-0daeb0fbadd1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.194545 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:43:11.194463    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls podName:55b1b828-311a-4e06-891f-0daeb0fbadd1 nodeName:}" failed. No retries permitted until 2026-11-01 09:43:11.694441725 +0000 UTC m=+4261.809314690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls") pod "maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2" (UID: "55b1b828-311a-4e06-891f-0daeb0fbadd1") : secret "maas-api-e2e-inf-b-b3d6bb-serving-cert" not found
Nov 01 09:43:11.195119 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.195091    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b1b828-311a-4e06-891f-0daeb0fbadd1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.197730 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.197702    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-metrics-certs\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.255621 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.255543    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwsp\" (UniqueName: \"kubernetes.io/projected/55b1b828-311a-4e06-891f-0daeb0fbadd1-kube-api-access-gwwsp\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.701521 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.701484    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.704378 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.704355    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:11.916572 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:11.916532    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:12.313424 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:12.313394    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"]
Nov 01 09:43:12.314132 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:43:12.314100    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b1b828_311a_4e06_891f_0daeb0fbadd1.slice/crio-dc8a4ebfa9a944209cb4c275b7fe423b9ddfe09c0d0d54093c5ee8d45c4d0d23 WatchSource:0}: Error finding container dc8a4ebfa9a944209cb4c275b7fe423b9ddfe09c0d0d54093c5ee8d45c4d0d23: Status 404 returned error can't find the container with id dc8a4ebfa9a944209cb4c275b7fe423b9ddfe09c0d0d54093c5ee8d45c4d0d23
Nov 01 09:43:13.120486 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:13.120447    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2" event={"ID":"55b1b828-311a-4e06-891f-0daeb0fbadd1","Type":"ContainerStarted","Data":"dc8a4ebfa9a944209cb4c275b7fe423b9ddfe09c0d0d54093c5ee8d45c4d0d23"}
Nov 01 09:43:14.157661 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:14.157616    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2" event={"ID":"55b1b828-311a-4e06-891f-0daeb0fbadd1","Type":"ContainerStarted","Data":"6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d"}
Nov 01 09:43:14.158249 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:14.157728    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:14.271141 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:14.271085    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2" podStartSLOduration=3.337992733 podStartE2EDuration="4.271070163s" totalImagesPullingTime="933.07743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:43:14.269880159 +0000 UTC m=+4264.384753128" watchObservedRunningTime="2026-11-01 09:43:14.271070163 +0000 UTC m=+4264.385943130"
Nov 01 09:43:20.171633 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:43:20.171555    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:43:31.025169 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:43:31.025130    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:44:01.024884 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:44:01.024835    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:44:31.024559 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:44:31.024516    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:45:00.222669 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.222588    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"]
Nov 01 09:45:00.226480 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.226463    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:00.249371 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.249337    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26j8f\" (UniqueName: \"kubernetes.io/projected/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-kube-api-access-26j8f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627\" (UID: \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:00.249564 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.249465    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627\" (UID: \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:00.251721 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.251694    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"]
Nov 01 09:45:00.256398 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.256364    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:00.261312 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.261282    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"]
Nov 01 09:45:00.267598 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.267573    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:00.295684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.295643    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 09:45:00.321563 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.321517    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"]
Nov 01 09:45:00.325897 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.325554    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"]
Nov 01 09:45:00.330468 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.330441    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"]
Nov 01 09:45:00.350558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.350522    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627\" (UID: \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:00.350764 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.350604    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b838b105-38c3-4633-ad29-98194114fa0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm\" (UID: \"b838b105-38c3-4633-ad29-98194114fa0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:00.350764 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.350652    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e55226-5dc0-4de0-bd93-2a9236c6aa85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx\" (UID: \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:00.350902 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.350821    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5w6\" (UniqueName: \"kubernetes.io/projected/69e55226-5dc0-4de0-bd93-2a9236c6aa85-kube-api-access-nq5w6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx\" (UID: \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:00.350902 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.350882    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26j8f\" (UniqueName: \"kubernetes.io/projected/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-kube-api-access-26j8f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627\" (UID: \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:00.351013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.350929    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snl6s\" (UniqueName: \"kubernetes.io/projected/b838b105-38c3-4633-ad29-98194114fa0c-kube-api-access-snl6s\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm\" (UID: \"b838b105-38c3-4633-ad29-98194114fa0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:00.351432 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.351408    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627\" (UID: \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:00.410584 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.410543    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26j8f\" (UniqueName: \"kubernetes.io/projected/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-kube-api-access-26j8f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627\" (UID: \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:00.452566 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.452518    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5w6\" (UniqueName: \"kubernetes.io/projected/69e55226-5dc0-4de0-bd93-2a9236c6aa85-kube-api-access-nq5w6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx\" (UID: \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:00.452777 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.452594    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snl6s\" (UniqueName: \"kubernetes.io/projected/b838b105-38c3-4633-ad29-98194114fa0c-kube-api-access-snl6s\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm\" (UID: \"b838b105-38c3-4633-ad29-98194114fa0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:00.452777 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.452671    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b838b105-38c3-4633-ad29-98194114fa0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm\" (UID: \"b838b105-38c3-4633-ad29-98194114fa0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:00.452777 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.452696    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e55226-5dc0-4de0-bd93-2a9236c6aa85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx\" (UID: \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:00.453447 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.453424    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e55226-5dc0-4de0-bd93-2a9236c6aa85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx\" (UID: \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:00.453567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.453543    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b838b105-38c3-4633-ad29-98194114fa0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm\" (UID: \"b838b105-38c3-4633-ad29-98194114fa0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:00.511607 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.511534    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snl6s\" (UniqueName: \"kubernetes.io/projected/b838b105-38c3-4633-ad29-98194114fa0c-kube-api-access-snl6s\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm\" (UID: \"b838b105-38c3-4633-ad29-98194114fa0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:00.511607 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.511541    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5w6\" (UniqueName: \"kubernetes.io/projected/69e55226-5dc0-4de0-bd93-2a9236c6aa85-kube-api-access-nq5w6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx\" (UID: \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:00.540087 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.540047    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:00.576361 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.576334    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:00.591410 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.591379    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:00.938191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.938159    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"]
Nov 01 09:45:00.938726 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:45:00.938703    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b11e6f0_b4ef_41c0_a5c6_57319ae3f7b0.slice/crio-15b17bf24d02dd16867bd97a95bae7f618fe323db797584f16a847dd9ff84d7d WatchSource:0}: Error finding container 15b17bf24d02dd16867bd97a95bae7f618fe323db797584f16a847dd9ff84d7d: Status 404 returned error can't find the container with id 15b17bf24d02dd16867bd97a95bae7f618fe323db797584f16a847dd9ff84d7d
Nov 01 09:45:00.983119 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:00.983090    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"]
Nov 01 09:45:00.983251 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:45:00.983228    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e55226_5dc0_4de0_bd93_2a9236c6aa85.slice/crio-f540d92611166d22a1026b51bce5c595baee07b72987f919a997e77bdc5792d6 WatchSource:0}: Error finding container f540d92611166d22a1026b51bce5c595baee07b72987f919a997e77bdc5792d6: Status 404 returned error can't find the container with id f540d92611166d22a1026b51bce5c595baee07b72987f919a997e77bdc5792d6
Nov 01 09:45:01.024773 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:01.024740    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:45:01.028235 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:01.028211    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"]
Nov 01 09:45:01.028523 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:45:01.028498    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb838b105_38c3_4633_ad29_98194114fa0c.slice/crio-10c592806f4f00b25b2a5468f05c87f162fda7708c151a2a04541cf7616d0762 WatchSource:0}: Error finding container 10c592806f4f00b25b2a5468f05c87f162fda7708c151a2a04541cf7616d0762: Status 404 returned error can't find the container with id 10c592806f4f00b25b2a5468f05c87f162fda7708c151a2a04541cf7616d0762
Nov 01 09:45:01.920935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:01.920844    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" event={"ID":"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0","Type":"ContainerStarted","Data":"8bebdcb24065070020b4c2abb3c1509eeca6f0e7dbd8e0313b3597506d06776f"}
Nov 01 09:45:01.920935 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:01.920906    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" event={"ID":"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0","Type":"ContainerStarted","Data":"15b17bf24d02dd16867bd97a95bae7f618fe323db797584f16a847dd9ff84d7d"}
Nov 01 09:45:01.922647 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:01.922622    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" event={"ID":"69e55226-5dc0-4de0-bd93-2a9236c6aa85","Type":"ContainerStarted","Data":"2fe17859b4a6c822f9905f85c43098b5fb639da40334500e5da013f42ff11602"}
Nov 01 09:45:01.922763 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:01.922674    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" event={"ID":"69e55226-5dc0-4de0-bd93-2a9236c6aa85","Type":"ContainerStarted","Data":"f540d92611166d22a1026b51bce5c595baee07b72987f919a997e77bdc5792d6"}
Nov 01 09:45:01.924295 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:01.924271    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" event={"ID":"b838b105-38c3-4633-ad29-98194114fa0c","Type":"ContainerStarted","Data":"9df6ebff6ca162e3bd2aec2a2c43f654112ed71612059f0303ea32535ed3d935"}
Nov 01 09:45:01.924395 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:01.924311    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" event={"ID":"b838b105-38c3-4633-ad29-98194114fa0c","Type":"ContainerStarted","Data":"10c592806f4f00b25b2a5468f05c87f162fda7708c151a2a04541cf7616d0762"}
Nov 01 09:45:02.032060 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.031988    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" podStartSLOduration=2.031975317 podStartE2EDuration="2.031975317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:02.030404198 +0000 UTC m=+4372.145277167" watchObservedRunningTime="2026-11-01 09:45:02.031975317 +0000 UTC m=+4372.146848284"
Nov 01 09:45:02.143139 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.143082    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" podStartSLOduration=2.143064977 podStartE2EDuration="2.143064977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:02.140189436 +0000 UTC m=+4372.255062403" watchObservedRunningTime="2026-11-01 09:45:02.143064977 +0000 UTC m=+4372.257937945"
Nov 01 09:45:02.251122 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.251068    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" podStartSLOduration=2.251051853 podStartE2EDuration="2.251051853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:02.249245623 +0000 UTC m=+4372.364118591" watchObservedRunningTime="2026-11-01 09:45:02.251051853 +0000 UTC m=+4372.365924822"
Nov 01 09:45:02.933972 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.933935    4931 generic.go:356] "Generic (PLEG): container finished" podID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerID="8bebdcb24065070020b4c2abb3c1509eeca6f0e7dbd8e0313b3597506d06776f" exitCode=60
Nov 01 09:45:02.934432 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.934045    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" event={"ID":"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0","Type":"ContainerDied","Data":"8bebdcb24065070020b4c2abb3c1509eeca6f0e7dbd8e0313b3597506d06776f"}
Nov 01 09:45:02.934432 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.934340    4931 scope.go:122] "RemoveContainer" containerID="8bebdcb24065070020b4c2abb3c1509eeca6f0e7dbd8e0313b3597506d06776f"
Nov 01 09:45:02.935769 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.935749    4931 generic.go:356] "Generic (PLEG): container finished" podID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerID="2fe17859b4a6c822f9905f85c43098b5fb639da40334500e5da013f42ff11602" exitCode=60
Nov 01 09:45:02.935875 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.935777    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" event={"ID":"69e55226-5dc0-4de0-bd93-2a9236c6aa85","Type":"ContainerDied","Data":"2fe17859b4a6c822f9905f85c43098b5fb639da40334500e5da013f42ff11602"}
Nov 01 09:45:02.936089 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.936071    4931 scope.go:122] "RemoveContainer" containerID="2fe17859b4a6c822f9905f85c43098b5fb639da40334500e5da013f42ff11602"
Nov 01 09:45:02.937622 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.937591    4931 generic.go:356] "Generic (PLEG): container finished" podID="b838b105-38c3-4633-ad29-98194114fa0c" containerID="9df6ebff6ca162e3bd2aec2a2c43f654112ed71612059f0303ea32535ed3d935" exitCode=60
Nov 01 09:45:02.937784 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.937695    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" event={"ID":"b838b105-38c3-4633-ad29-98194114fa0c","Type":"ContainerDied","Data":"9df6ebff6ca162e3bd2aec2a2c43f654112ed71612059f0303ea32535ed3d935"}
Nov 01 09:45:02.937934 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:02.937917    4931 scope.go:122] "RemoveContainer" containerID="9df6ebff6ca162e3bd2aec2a2c43f654112ed71612059f0303ea32535ed3d935"
Nov 01 09:45:03.947109 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:03.947063    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" event={"ID":"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0","Type":"ContainerStarted","Data":"2163712eca72d27468aed1075e0df6530cf10ae76496a5d49ca9327b8caf78a7"}
Nov 01 09:45:03.949287 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:03.949258    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" event={"ID":"69e55226-5dc0-4de0-bd93-2a9236c6aa85","Type":"ContainerStarted","Data":"eae9e5c52671b8e95d91943b7ab7a99f1a578fbc79b206e5a78eccccccf2a835"}
Nov 01 09:45:03.951491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:03.951463    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" event={"ID":"b838b105-38c3-4633-ad29-98194114fa0c","Type":"ContainerStarted","Data":"0930ac79aaa4b9d27e2ad3074718affd3d661ab27c925e7405c1c6958991ff68"}
Nov 01 09:45:04.961060 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.960994    4931 generic.go:356] "Generic (PLEG): container finished" podID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerID="eae9e5c52671b8e95d91943b7ab7a99f1a578fbc79b206e5a78eccccccf2a835" exitCode=60
Nov 01 09:45:04.961553 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.961072    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" event={"ID":"69e55226-5dc0-4de0-bd93-2a9236c6aa85","Type":"ContainerDied","Data":"eae9e5c52671b8e95d91943b7ab7a99f1a578fbc79b206e5a78eccccccf2a835"}
Nov 01 09:45:04.961553 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.961149    4931 scope.go:122] "RemoveContainer" containerID="2fe17859b4a6c822f9905f85c43098b5fb639da40334500e5da013f42ff11602"
Nov 01 09:45:04.961553 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.961450    4931 scope.go:122] "RemoveContainer" containerID="eae9e5c52671b8e95d91943b7ab7a99f1a578fbc79b206e5a78eccccccf2a835"
Nov 01 09:45:04.961772 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:04.961744    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx_odh-ai-gateway-infra(69e55226-5dc0-4de0-bd93-2a9236c6aa85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85"
Nov 01 09:45:04.963604 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.963582    4931 generic.go:356] "Generic (PLEG): container finished" podID="b838b105-38c3-4633-ad29-98194114fa0c" containerID="0930ac79aaa4b9d27e2ad3074718affd3d661ab27c925e7405c1c6958991ff68" exitCode=60
Nov 01 09:45:04.963713 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.963661    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" event={"ID":"b838b105-38c3-4633-ad29-98194114fa0c","Type":"ContainerDied","Data":"0930ac79aaa4b9d27e2ad3074718affd3d661ab27c925e7405c1c6958991ff68"}
Nov 01 09:45:04.963868 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.963851    4931 scope.go:122] "RemoveContainer" containerID="0930ac79aaa4b9d27e2ad3074718affd3d661ab27c925e7405c1c6958991ff68"
Nov 01 09:45:04.964162 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:04.964145    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm_odh-ai-gateway-infra(b838b105-38c3-4633-ad29-98194114fa0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" podUID="b838b105-38c3-4633-ad29-98194114fa0c"
Nov 01 09:45:04.965636 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.965615    4931 generic.go:356] "Generic (PLEG): container finished" podID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerID="2163712eca72d27468aed1075e0df6530cf10ae76496a5d49ca9327b8caf78a7" exitCode=60
Nov 01 09:45:04.965739 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.965682    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" event={"ID":"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0","Type":"ContainerDied","Data":"2163712eca72d27468aed1075e0df6530cf10ae76496a5d49ca9327b8caf78a7"}
Nov 01 09:45:04.965852 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.965840    4931 scope.go:122] "RemoveContainer" containerID="2163712eca72d27468aed1075e0df6530cf10ae76496a5d49ca9327b8caf78a7"
Nov 01 09:45:04.966078 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:04.966060    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627_odh-ai-gateway-infra(3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0"
Nov 01 09:45:04.975369 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.975350    4931 scope.go:122] "RemoveContainer" containerID="9df6ebff6ca162e3bd2aec2a2c43f654112ed71612059f0303ea32535ed3d935"
Nov 01 09:45:04.989798 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:04.989772    4931 scope.go:122] "RemoveContainer" containerID="8bebdcb24065070020b4c2abb3c1509eeca6f0e7dbd8e0313b3597506d06776f"
Nov 01 09:45:05.974056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:05.974013    4931 scope.go:122] "RemoveContainer" containerID="0930ac79aaa4b9d27e2ad3074718affd3d661ab27c925e7405c1c6958991ff68"
Nov 01 09:45:05.974511 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:05.974339    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm_odh-ai-gateway-infra(b838b105-38c3-4633-ad29-98194114fa0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" podUID="b838b105-38c3-4633-ad29-98194114fa0c"
Nov 01 09:45:05.975638 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:05.975613    4931 scope.go:122] "RemoveContainer" containerID="2163712eca72d27468aed1075e0df6530cf10ae76496a5d49ca9327b8caf78a7"
Nov 01 09:45:05.975926 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:05.975903    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627_odh-ai-gateway-infra(3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0"
Nov 01 09:45:05.977193 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:05.977176    4931 scope.go:122] "RemoveContainer" containerID="eae9e5c52671b8e95d91943b7ab7a99f1a578fbc79b206e5a78eccccccf2a835"
Nov 01 09:45:05.977349 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:05.977334    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx_odh-ai-gateway-infra(69e55226-5dc0-4de0-bd93-2a9236c6aa85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85"
Nov 01 09:45:15.040755 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:15.040723    4931 scope.go:122] "RemoveContainer" containerID="2163712eca72d27468aed1075e0df6530cf10ae76496a5d49ca9327b8caf78a7"
Nov 01 09:45:16.055437 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:16.055402    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" event={"ID":"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0","Type":"ContainerStarted","Data":"7fdcebc312ee1d75bd0e0d879818f692db741568941abeed2b2b8be911a10c3d"}
Nov 01 09:45:17.066162 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:17.066125    4931 generic.go:356] "Generic (PLEG): container finished" podID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerID="7fdcebc312ee1d75bd0e0d879818f692db741568941abeed2b2b8be911a10c3d" exitCode=60
Nov 01 09:45:17.066589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:17.066207    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" event={"ID":"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0","Type":"ContainerDied","Data":"7fdcebc312ee1d75bd0e0d879818f692db741568941abeed2b2b8be911a10c3d"}
Nov 01 09:45:17.066589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:17.066283    4931 scope.go:122] "RemoveContainer" containerID="2163712eca72d27468aed1075e0df6530cf10ae76496a5d49ca9327b8caf78a7"
Nov 01 09:45:17.066766 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:17.066719    4931 scope.go:122] "RemoveContainer" containerID="7fdcebc312ee1d75bd0e0d879818f692db741568941abeed2b2b8be911a10c3d"
Nov 01 09:45:17.067089 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:17.067064    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627_odh-ai-gateway-infra(3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0"
Nov 01 09:45:17.178742 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:17.178706    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"]
Nov 01 09:45:18.040820 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.040782    4931 scope.go:122] "RemoveContainer" containerID="0930ac79aaa4b9d27e2ad3074718affd3d661ab27c925e7405c1c6958991ff68"
Nov 01 09:45:18.041063 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.040857    4931 scope.go:122] "RemoveContainer" containerID="eae9e5c52671b8e95d91943b7ab7a99f1a578fbc79b206e5a78eccccccf2a835"
Nov 01 09:45:18.302603 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.302578    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:18.422620 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.422581    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-maas-api-service-ca\") pod \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\" (UID: \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\") "
Nov 01 09:45:18.422819 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.422649    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-kube-api-access-26j8f\" (UniqueName: \"kubernetes.io/projected/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-kube-api-access-26j8f\") pod \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\" (UID: \"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0\") "
Nov 01 09:45:18.422963 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.422939    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-maas-api-service-ca" pod "3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" (UID: "3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:18.461093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.459808    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-kube-api-access-26j8f" pod "3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" (UID: "3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0"). InnerVolumeSpecName "kube-api-access-26j8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:18.524901 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.524852    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:18.524901 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:18.524893    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26j8f\" (UniqueName: \"kubernetes.io/projected/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0-kube-api-access-26j8f\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:19.105177 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.100524    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627" event={"ID":"3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0","Type":"ContainerDied","Data":"15b17bf24d02dd16867bd97a95bae7f618fe323db797584f16a847dd9ff84d7d"}
Nov 01 09:45:19.105177 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.100602    4931 scope.go:122] "RemoveContainer" containerID="7fdcebc312ee1d75bd0e0d879818f692db741568941abeed2b2b8be911a10c3d"
Nov 01 09:45:19.105177 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.100748    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"
Nov 01 09:45:19.110225 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.110196    4931 generic.go:356] "Generic (PLEG): container finished" podID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerID="85601b8b41b0b1bdd143deb9454bb7374650bd1a1f8b6a8f7e151bf056870c8c" exitCode=60
Nov 01 09:45:19.110372 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.110279    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" event={"ID":"69e55226-5dc0-4de0-bd93-2a9236c6aa85","Type":"ContainerDied","Data":"85601b8b41b0b1bdd143deb9454bb7374650bd1a1f8b6a8f7e151bf056870c8c"}
Nov 01 09:45:19.110796 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.110772    4931 scope.go:122] "RemoveContainer" containerID="85601b8b41b0b1bdd143deb9454bb7374650bd1a1f8b6a8f7e151bf056870c8c"
Nov 01 09:45:19.111179 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:19.111152    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx_odh-ai-gateway-infra(69e55226-5dc0-4de0-bd93-2a9236c6aa85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85"
Nov 01 09:45:19.113116 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.113092    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" event={"ID":"b838b105-38c3-4633-ad29-98194114fa0c","Type":"ContainerStarted","Data":"114988d5940871d61e27c980cd1ea096f8cca3429c74e594c8cc041752c7ddb2"}
Nov 01 09:45:19.127915 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.127890    4931 scope.go:122] "RemoveContainer" containerID="eae9e5c52671b8e95d91943b7ab7a99f1a578fbc79b206e5a78eccccccf2a835"
Nov 01 09:45:19.268289 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.268254    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"]
Nov 01 09:45:19.273549 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:19.273521    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5b96ac-29892105-22627"]
Nov 01 09:45:20.048875 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.048834    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" path="/var/lib/kubelet/pods/3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0/volumes"
Nov 01 09:45:20.128800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.128758    4931 generic.go:356] "Generic (PLEG): container finished" podID="b838b105-38c3-4633-ad29-98194114fa0c" containerID="114988d5940871d61e27c980cd1ea096f8cca3429c74e594c8cc041752c7ddb2" exitCode=60
Nov 01 09:45:20.128800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.128785    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" event={"ID":"b838b105-38c3-4633-ad29-98194114fa0c","Type":"ContainerDied","Data":"114988d5940871d61e27c980cd1ea096f8cca3429c74e594c8cc041752c7ddb2"}
Nov 01 09:45:20.129056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.128859    4931 scope.go:122] "RemoveContainer" containerID="0930ac79aaa4b9d27e2ad3074718affd3d661ab27c925e7405c1c6958991ff68"
Nov 01 09:45:20.129383 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.129359    4931 scope.go:122] "RemoveContainer" containerID="114988d5940871d61e27c980cd1ea096f8cca3429c74e594c8cc041752c7ddb2"
Nov 01 09:45:20.129750 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:20.129725    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm_odh-ai-gateway-infra(b838b105-38c3-4633-ad29-98194114fa0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" podUID="b838b105-38c3-4633-ad29-98194114fa0c"
Nov 01 09:45:20.388980 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.388921    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"]
Nov 01 09:45:20.505858 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.505600    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"]
Nov 01 09:45:20.560495 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.560465    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:20.659993 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.659893    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e55226-5dc0-4de0-bd93-2a9236c6aa85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e55226-5dc0-4de0-bd93-2a9236c6aa85-maas-api-service-ca\") pod \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\" (UID: \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\") "
Nov 01 09:45:20.659993 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.659982    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e55226-5dc0-4de0-bd93-2a9236c6aa85-kube-api-access-nq5w6\" (UniqueName: \"kubernetes.io/projected/69e55226-5dc0-4de0-bd93-2a9236c6aa85-kube-api-access-nq5w6\") pod \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\" (UID: \"69e55226-5dc0-4de0-bd93-2a9236c6aa85\") "
Nov 01 09:45:20.660369 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.660343    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e55226-5dc0-4de0-bd93-2a9236c6aa85-maas-api-service-ca" pod "69e55226-5dc0-4de0-bd93-2a9236c6aa85" (UID: "69e55226-5dc0-4de0-bd93-2a9236c6aa85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:20.660542 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.660522    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e55226-5dc0-4de0-bd93-2a9236c6aa85-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:20.663056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.662997    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e55226-5dc0-4de0-bd93-2a9236c6aa85-kube-api-access-nq5w6" pod "69e55226-5dc0-4de0-bd93-2a9236c6aa85" (UID: "69e55226-5dc0-4de0-bd93-2a9236c6aa85"). InnerVolumeSpecName "kube-api-access-nq5w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:20.761661 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:20.761621    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq5w6\" (UniqueName: \"kubernetes.io/projected/69e55226-5dc0-4de0-bd93-2a9236c6aa85-kube-api-access-nq5w6\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:21.151893 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.151862    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"
Nov 01 09:45:21.152371 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.151903    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx" event={"ID":"69e55226-5dc0-4de0-bd93-2a9236c6aa85","Type":"ContainerDied","Data":"f540d92611166d22a1026b51bce5c595baee07b72987f919a997e77bdc5792d6"}
Nov 01 09:45:21.152371 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.151992    4931 scope.go:122] "RemoveContainer" containerID="85601b8b41b0b1bdd143deb9454bb7374650bd1a1f8b6a8f7e151bf056870c8c"
Nov 01 09:45:21.314427 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.314370    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"]
Nov 01 09:45:21.317747 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.317719    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a87a1d-29892105-p9nrx"]
Nov 01 09:45:21.367038 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.366998    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:21.369782 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.369757    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b838b105-38c3-4633-ad29-98194114fa0c-kube-api-access-snl6s\" (UniqueName: \"kubernetes.io/projected/b838b105-38c3-4633-ad29-98194114fa0c-kube-api-access-snl6s\") pod \"b838b105-38c3-4633-ad29-98194114fa0c\" (UID: \"b838b105-38c3-4633-ad29-98194114fa0c\") "
Nov 01 09:45:21.369936 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.369810    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b838b105-38c3-4633-ad29-98194114fa0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b838b105-38c3-4633-ad29-98194114fa0c-maas-api-service-ca\") pod \"b838b105-38c3-4633-ad29-98194114fa0c\" (UID: \"b838b105-38c3-4633-ad29-98194114fa0c\") "
Nov 01 09:45:21.370295 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.370271    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b838b105-38c3-4633-ad29-98194114fa0c-maas-api-service-ca" pod "b838b105-38c3-4633-ad29-98194114fa0c" (UID: "b838b105-38c3-4633-ad29-98194114fa0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:21.372146 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.372123    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b838b105-38c3-4633-ad29-98194114fa0c-kube-api-access-snl6s" pod "b838b105-38c3-4633-ad29-98194114fa0c" (UID: "b838b105-38c3-4633-ad29-98194114fa0c"). InnerVolumeSpecName "kube-api-access-snl6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:21.471338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.471247    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snl6s\" (UniqueName: \"kubernetes.io/projected/b838b105-38c3-4633-ad29-98194114fa0c-kube-api-access-snl6s\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:21.471338 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:21.471274    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b838b105-38c3-4633-ad29-98194114fa0c-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:22.048579 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:22.048545    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" path="/var/lib/kubelet/pods/69e55226-5dc0-4de0-bd93-2a9236c6aa85/volumes"
Nov 01 09:45:22.163650 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:22.163623    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"
Nov 01 09:45:22.164101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:22.163680    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm" event={"ID":"b838b105-38c3-4633-ad29-98194114fa0c","Type":"ContainerDied","Data":"10c592806f4f00b25b2a5468f05c87f162fda7708c151a2a04541cf7616d0762"}
Nov 01 09:45:22.164101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:22.163745    4931 scope.go:122] "RemoveContainer" containerID="114988d5940871d61e27c980cd1ea096f8cca3429c74e594c8cc041752c7ddb2"
Nov 01 09:45:22.329482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:22.329398    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"]
Nov 01 09:45:22.378178 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:22.378142    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-65aeef-29892105-bbqdm"]
Nov 01 09:45:24.049297 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:24.049265    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b838b105-38c3-4633-ad29-98194114fa0c" path="/var/lib/kubelet/pods/b838b105-38c3-4633-ad29-98194114fa0c/volumes"
Nov 01 09:45:31.023893 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:31.023850    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:45:45.682747 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:45.682716    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"]
Nov 01 09:45:45.683246 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:45.682970    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt" podUID="f598d227-7b48-4f40-aa74-c0421b67266e" containerName="maas-api" containerID="cri-o://3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009" gracePeriod=30
Nov 01 09:45:46.022512 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.022487    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:45:46.175051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.168599    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-metrics-certs\") pod \"f598d227-7b48-4f40-aa74-c0421b67266e\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") "
Nov 01 09:45:46.175051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.168658    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls\") pod \"f598d227-7b48-4f40-aa74-c0421b67266e\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") "
Nov 01 09:45:46.175051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.168697    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f598d227-7b48-4f40-aa74-c0421b67266e-kube-api-access-vrtlw\" (UniqueName: \"kubernetes.io/projected/f598d227-7b48-4f40-aa74-c0421b67266e-kube-api-access-vrtlw\") pod \"f598d227-7b48-4f40-aa74-c0421b67266e\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") "
Nov 01 09:45:46.175051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.168762    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f598d227-7b48-4f40-aa74-c0421b67266e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f598d227-7b48-4f40-aa74-c0421b67266e-openshift-service-ca\") pod \"f598d227-7b48-4f40-aa74-c0421b67266e\" (UID: \"f598d227-7b48-4f40-aa74-c0421b67266e\") "
Nov 01 09:45:46.175051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.169940    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f598d227-7b48-4f40-aa74-c0421b67266e-openshift-service-ca" pod "f598d227-7b48-4f40-aa74-c0421b67266e" (UID: "f598d227-7b48-4f40-aa74-c0421b67266e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:45:46.179046 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.178785    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-metrics-certs" pod "f598d227-7b48-4f40-aa74-c0421b67266e" (UID: "f598d227-7b48-4f40-aa74-c0421b67266e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:45:46.184186 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.179798    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls" pod "f598d227-7b48-4f40-aa74-c0421b67266e" (UID: "f598d227-7b48-4f40-aa74-c0421b67266e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:45:46.184186 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.183850    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f598d227-7b48-4f40-aa74-c0421b67266e-kube-api-access-vrtlw" pod "f598d227-7b48-4f40-aa74-c0421b67266e" (UID: "f598d227-7b48-4f40-aa74-c0421b67266e"). InnerVolumeSpecName "kube-api-access-vrtlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:45:46.271325 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.271062    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f598d227-7b48-4f40-aa74-c0421b67266e-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:46.271325 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.271096    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:46.271325 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.271118    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f598d227-7b48-4f40-aa74-c0421b67266e-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:46.271325 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.271128    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrtlw\" (UniqueName: \"kubernetes.io/projected/f598d227-7b48-4f40-aa74-c0421b67266e-kube-api-access-vrtlw\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:45:46.378620 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.378575    4931 generic.go:356] "Generic (PLEG): container finished" podID="f598d227-7b48-4f40-aa74-c0421b67266e" containerID="3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009" exitCode=0
Nov 01 09:45:46.378620 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.378621    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"
Nov 01 09:45:46.378857 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.378660    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt" event={"ID":"f598d227-7b48-4f40-aa74-c0421b67266e","Type":"ContainerDied","Data":"3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009"}
Nov 01 09:45:46.378857 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.378715    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt" event={"ID":"f598d227-7b48-4f40-aa74-c0421b67266e","Type":"ContainerDied","Data":"ddd9c58a25dcec0477546b2cb56496efc250053763b4ffb8a949f6ee763418df"}
Nov 01 09:45:46.378857 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.378745    4931 scope.go:122] "RemoveContainer" containerID="3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009"
Nov 01 09:45:46.389975 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.389959    4931 scope.go:122] "RemoveContainer" containerID="3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009"
Nov 01 09:45:46.390286 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:45:46.390263    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009\": container with ID starting with 3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009 not found: ID does not exist" containerID="3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009"
Nov 01 09:45:46.390341 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.390291    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009"} err="failed to get container status \"3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009\": rpc error: code = NotFound desc = could not find container \"3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009\": container with ID starting with 3a96680ee653832835acb0d49b9dca99a5fc1409ab7fb9c31e1b6c48905ca009 not found: ID does not exist"
Nov 01 09:45:46.538062 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.537958    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"]
Nov 01 09:45:46.543966 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:46.543944    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49f18f-5b6b56d4d4-kddbt"]
Nov 01 09:45:48.053370 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:45:48.053324    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f598d227-7b48-4f40-aa74-c0421b67266e" path="/var/lib/kubelet/pods/f598d227-7b48-4f40-aa74-c0421b67266e/volumes"
Nov 01 09:46:01.022786 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:46:01.022741    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:46:10.799309 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:10.799270    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rvlsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 09:46:10.799915 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:10.799882    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:46:31.022924 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:46:31.022842    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:46:46.430300 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430268    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"]
Nov 01 09:46:46.430831 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430818    4931 cpu_manager.go:414] "containerMap: removing container" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:46:46.430879 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430831    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:46:46.430879 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430844    4931 cpu_manager.go:414] "containerMap: removing container" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:46:46.430879 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430849    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:46:46.430879 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430857    4931 cpu_manager.go:414] "containerMap: removing container" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:46:46.430879 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430861    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:46:46.430879 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430880    4931 cpu_manager.go:414] "containerMap: removing container" podUID="f598d227-7b48-4f40-aa74-c0421b67266e" containerName="maas-api"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430886    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f598d227-7b48-4f40-aa74-c0421b67266e" containerName="maas-api"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430898    4931 cpu_manager.go:414] "containerMap: removing container" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430903    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430914    4931 cpu_manager.go:414] "containerMap: removing container" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430918    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430932    4931 cpu_manager.go:414] "containerMap: removing container" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.430937    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.431010    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.431017    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.431041    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.431053    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.431061    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.431073    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f598d227-7b48-4f40-aa74-c0421b67266e" containerName="maas-api"
Nov 01 09:46:46.431082 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.431083    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:46:46.431519 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.431093    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:46:46.434670 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.434653    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.456142 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.456119    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-metrics-certs\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.456274 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.456187    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clq86\" (UniqueName: \"kubernetes.io/projected/4b639a78-6674-4a33-8e9a-56f0e84e70c0-kube-api-access-clq86\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.456339 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.456319    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.456412 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.456392    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b639a78-6674-4a33-8e9a-56f0e84e70c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.484686 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.484661    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3710de-serving-cert\""
Nov 01 09:46:46.536011 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.535982    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"]
Nov 01 09:46:46.557139 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.557111    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-metrics-certs\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.557312 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.557205    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clq86\" (UniqueName: \"kubernetes.io/projected/4b639a78-6674-4a33-8e9a-56f0e84e70c0-kube-api-access-clq86\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.557380 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.557317    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.557438 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.557409    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b639a78-6674-4a33-8e9a-56f0e84e70c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.557499 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:46:46.557472    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-serving-cert: secret "maas-api-e2e-inf-a-3710de-serving-cert" not found
Nov 01 09:46:46.557575 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:46:46.557561    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls podName:4b639a78-6674-4a33-8e9a-56f0e84e70c0 nodeName:}" failed. No retries permitted until 2026-11-01 09:46:47.057540772 +0000 UTC m=+4477.172413718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls") pod "maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p" (UID: "4b639a78-6674-4a33-8e9a-56f0e84e70c0") : secret "maas-api-e2e-inf-a-3710de-serving-cert" not found
Nov 01 09:46:46.558089 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.558067    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b639a78-6674-4a33-8e9a-56f0e84e70c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.559895 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.559871    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-metrics-certs\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:46.618081 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:46.618048    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clq86\" (UniqueName: \"kubernetes.io/projected/4b639a78-6674-4a33-8e9a-56f0e84e70c0-kube-api-access-clq86\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:47.064444 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:47.064402    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:47.067158 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:47.067132    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:47.347914 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:47.347843    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:47.722819 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:47.722790    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"]
Nov 01 09:46:47.723641 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:46:47.723615    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b639a78_6674_4a33_8e9a_56f0e84e70c0.slice/crio-2996ef73502c5524fc32c72f09e01f5503ee4cb9ddc0fa04bc7c196cf138ddf8 WatchSource:0}: Error finding container 2996ef73502c5524fc32c72f09e01f5503ee4cb9ddc0fa04bc7c196cf138ddf8: Status 404 returned error can't find the container with id 2996ef73502c5524fc32c72f09e01f5503ee4cb9ddc0fa04bc7c196cf138ddf8
Nov 01 09:46:47.725430 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:47.725409    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 09:46:47.802746 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:47.802716    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p" event={"ID":"4b639a78-6674-4a33-8e9a-56f0e84e70c0","Type":"ContainerStarted","Data":"2996ef73502c5524fc32c72f09e01f5503ee4cb9ddc0fa04bc7c196cf138ddf8"}
Nov 01 09:46:48.810818 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:48.810785    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p" event={"ID":"4b639a78-6674-4a33-8e9a-56f0e84e70c0","Type":"ContainerStarted","Data":"47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f"}
Nov 01 09:46:48.811259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:48.810870    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:46:48.956169 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:48.956081    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p" podStartSLOduration=2.01118155 podStartE2EDuration="2.956067273s" totalImagesPullingTime="944.885723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:46:48.954729556 +0000 UTC m=+4479.069602523" watchObservedRunningTime="2026-11-01 09:46:48.956067273 +0000 UTC m=+4479.070940243"
Nov 01 09:46:54.822353 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:46:54.822317    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:47:01.023744 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:47:01.023699    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:47:10.839243 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:10.839121    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:47:10.851323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:10.843390    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:47:10.864639 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:10.864622    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:47:10.868456 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:10.868436    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:47:24.297838 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.297804    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"]
Nov 01 09:47:24.298514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.298012    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj" podUID="4753eca6-0326-4eaa-b4ca-63e2a5caafb4" containerName="maas-api" containerID="cri-o://5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa" gracePeriod=30
Nov 01 09:47:24.628817 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.628782    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:47:24.718050 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.717406    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-openshift-service-ca\") pod \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") "
Nov 01 09:47:24.718050 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.717486    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-metrics-certs\") pod \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") "
Nov 01 09:47:24.718050 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.717570    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-kube-api-access-skvx8\" (UniqueName: \"kubernetes.io/projected/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-kube-api-access-skvx8\") pod \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") "
Nov 01 09:47:24.718050 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.717597    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-maas-api-tls\") pod \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\" (UID: \"4753eca6-0326-4eaa-b4ca-63e2a5caafb4\") "
Nov 01 09:47:24.726153 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.726115    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-openshift-service-ca" pod "4753eca6-0326-4eaa-b4ca-63e2a5caafb4" (UID: "4753eca6-0326-4eaa-b4ca-63e2a5caafb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:47:24.732180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.732142    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-maas-api-tls" pod "4753eca6-0326-4eaa-b4ca-63e2a5caafb4" (UID: "4753eca6-0326-4eaa-b4ca-63e2a5caafb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:47:24.733211 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.733164    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-metrics-certs" pod "4753eca6-0326-4eaa-b4ca-63e2a5caafb4" (UID: "4753eca6-0326-4eaa-b4ca-63e2a5caafb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:47:24.738541 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.738503    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-kube-api-access-skvx8" pod "4753eca6-0326-4eaa-b4ca-63e2a5caafb4" (UID: "4753eca6-0326-4eaa-b4ca-63e2a5caafb4"). InnerVolumeSpecName "kube-api-access-skvx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:24.818927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.818885    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:24.818927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.818915    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skvx8\" (UniqueName: \"kubernetes.io/projected/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-kube-api-access-skvx8\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:24.818927 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.818930    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:24.819242 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:24.818945    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4753eca6-0326-4eaa-b4ca-63e2a5caafb4-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:25.067325 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.067286    4931 generic.go:356] "Generic (PLEG): container finished" podID="4753eca6-0326-4eaa-b4ca-63e2a5caafb4" containerID="5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa" exitCode=0
Nov 01 09:47:25.067515 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.067339    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"
Nov 01 09:47:25.067515 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.067359    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj" event={"ID":"4753eca6-0326-4eaa-b4ca-63e2a5caafb4","Type":"ContainerDied","Data":"5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa"}
Nov 01 09:47:25.067515 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.067415    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj" event={"ID":"4753eca6-0326-4eaa-b4ca-63e2a5caafb4","Type":"ContainerDied","Data":"44d6eaad5a3b5fe7f6798ec4e9746f86fefca29693e2d846b86a991d6bb71419"}
Nov 01 09:47:25.067515 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.067436    4931 scope.go:122] "RemoveContainer" containerID="5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa"
Nov 01 09:47:25.078789 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.078767    4931 scope.go:122] "RemoveContainer" containerID="5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa"
Nov 01 09:47:25.079096 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:47:25.079079    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa\": container with ID starting with 5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa not found: ID does not exist" containerID="5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa"
Nov 01 09:47:25.079146 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.079105    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa"} err="failed to get container status \"5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa\": rpc error: code = NotFound desc = could not find container \"5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa\": container with ID starting with 5760079ca2dd61624b140324cd3122da34a99d705dd525ea666e521a721511aa not found: ID does not exist"
Nov 01 09:47:25.226469 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.226440    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"]
Nov 01 09:47:25.294486 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:25.294450    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cddc5b-84767f95b5-hszlj"]
Nov 01 09:47:26.049106 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:26.049075    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4753eca6-0326-4eaa-b4ca-63e2a5caafb4" path="/var/lib/kubelet/pods/4753eca6-0326-4eaa-b4ca-63e2a5caafb4/volumes"
Nov 01 09:47:31.022928 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:47:31.022885    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:47:35.691618 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:35.691578    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"]
Nov 01 09:47:35.692133 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:35.691825    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc" podUID="310768c0-2031-445b-8014-d7c4ecef7fb4" containerName="maas-api" containerID="cri-o://30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78" gracePeriod=30
Nov 01 09:47:35.772469 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:47:35.772423    4931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310768c0_2031_445b_8014_d7c4ecef7fb4.slice/crio-conmon-30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 09:47:36.052506 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.052476    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:47:36.156376 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.156341    4931 generic.go:356] "Generic (PLEG): container finished" podID="310768c0-2031-445b-8014-d7c4ecef7fb4" containerID="30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78" exitCode=0
Nov 01 09:47:36.156787 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.156770    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"
Nov 01 09:47:36.157137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.157108    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc" event={"ID":"310768c0-2031-445b-8014-d7c4ecef7fb4","Type":"ContainerDied","Data":"30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78"}
Nov 01 09:47:36.157310 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.157291    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc" event={"ID":"310768c0-2031-445b-8014-d7c4ecef7fb4","Type":"ContainerDied","Data":"47650dbdb3589f93afc26d8cf32308911dc76a7f198c9a2d7c69fe534369f7c7"}
Nov 01 09:47:36.157430 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.157418    4931 scope.go:122] "RemoveContainer" containerID="30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78"
Nov 01 09:47:36.171894 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.171872    4931 scope.go:122] "RemoveContainer" containerID="30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78"
Nov 01 09:47:36.177713 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:47:36.177532    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78\": container with ID starting with 30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78 not found: ID does not exist" containerID="30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78"
Nov 01 09:47:36.177713 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.177670    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78"} err="failed to get container status \"30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78\": rpc error: code = NotFound desc = could not find container \"30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78\": container with ID starting with 30465162bed07eab4af6dd703610ebbc7f02f9bfcb6ccf4dd0bf8da5c8e13d78 not found: ID does not exist"
Nov 01 09:47:36.183144 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.183115    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310768c0-2031-445b-8014-d7c4ecef7fb4-kube-api-access-4zwrc\" (UniqueName: \"kubernetes.io/projected/310768c0-2031-445b-8014-d7c4ecef7fb4-kube-api-access-4zwrc\") pod \"310768c0-2031-445b-8014-d7c4ecef7fb4\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") "
Nov 01 09:47:36.183284 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.183218    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310768c0-2031-445b-8014-d7c4ecef7fb4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310768c0-2031-445b-8014-d7c4ecef7fb4-openshift-service-ca\") pod \"310768c0-2031-445b-8014-d7c4ecef7fb4\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") "
Nov 01 09:47:36.183284 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.183253    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-metrics-certs\") pod \"310768c0-2031-445b-8014-d7c4ecef7fb4\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") "
Nov 01 09:47:36.183447 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.183426    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-maas-api-tls\") pod \"310768c0-2031-445b-8014-d7c4ecef7fb4\" (UID: \"310768c0-2031-445b-8014-d7c4ecef7fb4\") "
Nov 01 09:47:36.184179 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.184156    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310768c0-2031-445b-8014-d7c4ecef7fb4-openshift-service-ca" pod "310768c0-2031-445b-8014-d7c4ecef7fb4" (UID: "310768c0-2031-445b-8014-d7c4ecef7fb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:47:36.187390 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.187358    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-metrics-certs" pod "310768c0-2031-445b-8014-d7c4ecef7fb4" (UID: "310768c0-2031-445b-8014-d7c4ecef7fb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:47:36.190420 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.190395    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310768c0-2031-445b-8014-d7c4ecef7fb4-kube-api-access-4zwrc" pod "310768c0-2031-445b-8014-d7c4ecef7fb4" (UID: "310768c0-2031-445b-8014-d7c4ecef7fb4"). InnerVolumeSpecName "kube-api-access-4zwrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:36.194195 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.194168    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-maas-api-tls" pod "310768c0-2031-445b-8014-d7c4ecef7fb4" (UID: "310768c0-2031-445b-8014-d7c4ecef7fb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:47:36.285920 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.285357    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310768c0-2031-445b-8014-d7c4ecef7fb4-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:36.285920 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.285396    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:36.285920 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.285410    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310768c0-2031-445b-8014-d7c4ecef7fb4-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:36.285920 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.285424    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zwrc\" (UniqueName: \"kubernetes.io/projected/310768c0-2031-445b-8014-d7c4ecef7fb4-kube-api-access-4zwrc\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:36.579923 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.579831    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"]
Nov 01 09:47:36.622884 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:36.622848    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c81a9d-84f757d9d7-j9wxc"]
Nov 01 09:47:38.048246 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:38.048214    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310768c0-2031-445b-8014-d7c4ecef7fb4" path="/var/lib/kubelet/pods/310768c0-2031-445b-8014-d7c4ecef7fb4/volumes"
Nov 01 09:47:46.564331 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.564298    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"]
Nov 01 09:47:46.564802 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.564543    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc" podUID="93d9f2ae-4845-4ba9-82b1-9f926b054e35" containerName="maas-api" containerID="cri-o://254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c" gracePeriod=30
Nov 01 09:47:46.685202 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.685155    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc" podUID="93d9f2ae-4845-4ba9-82b1-9f926b054e35" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.129:8443/health\": dial tcp 10.133.0.129:8443: connect: connection refused"
Nov 01 09:47:46.854906 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.854876    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:47:46.905106 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.905070    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d9f2ae-4845-4ba9-82b1-9f926b054e35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d9f2ae-4845-4ba9-82b1-9f926b054e35-openshift-service-ca\") pod \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") "
Nov 01 09:47:46.905305 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.905276    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-metrics-certs\") pod \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") "
Nov 01 09:47:46.905375 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.905311    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d9f2ae-4845-4ba9-82b1-9f926b054e35-kube-api-access-gct8z\" (UniqueName: \"kubernetes.io/projected/93d9f2ae-4845-4ba9-82b1-9f926b054e35-kube-api-access-gct8z\") pod \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") "
Nov 01 09:47:46.905441 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.905375    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls\") pod \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\" (UID: \"93d9f2ae-4845-4ba9-82b1-9f926b054e35\") "
Nov 01 09:47:46.905850 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.905492    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d9f2ae-4845-4ba9-82b1-9f926b054e35-openshift-service-ca" pod "93d9f2ae-4845-4ba9-82b1-9f926b054e35" (UID: "93d9f2ae-4845-4ba9-82b1-9f926b054e35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:47:46.905850 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.905787    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d9f2ae-4845-4ba9-82b1-9f926b054e35-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:46.908455 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.908424    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-metrics-certs" pod "93d9f2ae-4845-4ba9-82b1-9f926b054e35" (UID: "93d9f2ae-4845-4ba9-82b1-9f926b054e35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:47:46.908675 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.908650    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d9f2ae-4845-4ba9-82b1-9f926b054e35-kube-api-access-gct8z" pod "93d9f2ae-4845-4ba9-82b1-9f926b054e35" (UID: "93d9f2ae-4845-4ba9-82b1-9f926b054e35"). InnerVolumeSpecName "kube-api-access-gct8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:47:46.908903 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:46.908867    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls" pod "93d9f2ae-4845-4ba9-82b1-9f926b054e35" (UID: "93d9f2ae-4845-4ba9-82b1-9f926b054e35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:47:47.006618 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.006587    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:47.006618 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.006617    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gct8z\" (UniqueName: \"kubernetes.io/projected/93d9f2ae-4845-4ba9-82b1-9f926b054e35-kube-api-access-gct8z\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:47.006813 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.006629    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9f2ae-4845-4ba9-82b1-9f926b054e35-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:47:47.241096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.241019    4931 generic.go:356] "Generic (PLEG): container finished" podID="93d9f2ae-4845-4ba9-82b1-9f926b054e35" containerID="254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c" exitCode=0
Nov 01 09:47:47.241096 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.241093    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"
Nov 01 09:47:47.241344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.241062    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc" event={"ID":"93d9f2ae-4845-4ba9-82b1-9f926b054e35","Type":"ContainerDied","Data":"254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c"}
Nov 01 09:47:47.241344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.241200    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc" event={"ID":"93d9f2ae-4845-4ba9-82b1-9f926b054e35","Type":"ContainerDied","Data":"928f185c55d036d3d990ea119d5bd30bec322b66470a27ea671cd62edb1341ed"}
Nov 01 09:47:47.241344 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.241232    4931 scope.go:122] "RemoveContainer" containerID="254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c"
Nov 01 09:47:47.253764 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.253744    4931 scope.go:122] "RemoveContainer" containerID="254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c"
Nov 01 09:47:47.254073 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:47:47.254051    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c\": container with ID starting with 254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c not found: ID does not exist" containerID="254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c"
Nov 01 09:47:47.254147 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.254082    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c"} err="failed to get container status \"254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c\": rpc error: code = NotFound desc = could not find container \"254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c\": container with ID starting with 254ad4cedf773a9530a3048df6d73abd279be001239b28bf48afde34045d8c8c not found: ID does not exist"
Nov 01 09:47:47.409571 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.409539    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"]
Nov 01 09:47:47.418384 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:47.418347    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a53f2-8d77f9448-hs7wc"]
Nov 01 09:47:48.048341 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:47:48.048298    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d9f2ae-4845-4ba9-82b1-9f926b054e35" path="/var/lib/kubelet/pods/93d9f2ae-4845-4ba9-82b1-9f926b054e35/volumes"
Nov 01 09:48:01.025423 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:48:01.025331    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:48:12.369253 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.369219    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"]
Nov 01 09:48:12.369753 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.369488    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x" podUID="def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" containerName="maas-api" containerID="cri-o://e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7" gracePeriod=30
Nov 01 09:48:12.687281 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.687251    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:48:12.719960 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.719922    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-metrics-certs\") pod \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") "
Nov 01 09:48:12.720167 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.720055    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-kube-api-access-9bpsf\" (UniqueName: \"kubernetes.io/projected/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-kube-api-access-9bpsf\") pod \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") "
Nov 01 09:48:12.720167 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.720149    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls\") pod \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") "
Nov 01 09:48:12.720377 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.720312    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-openshift-service-ca\") pod \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\" (UID: \"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2\") "
Nov 01 09:48:12.720724 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.720688    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-openshift-service-ca" pod "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" (UID: "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:48:12.721106 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.721083    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:12.722542 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.722512    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-metrics-certs" pod "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" (UID: "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:48:12.722648 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.722533    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-kube-api-access-9bpsf" pod "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" (UID: "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2"). InnerVolumeSpecName "kube-api-access-9bpsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:48:12.722933 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.722914    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls" pod "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" (UID: "def75dd7-bd4b-4b93-a463-a4fbbbeb85a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:48:12.825014 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.824978    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:12.825014 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.825010    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:12.825014 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:12.825040    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpsf\" (UniqueName: \"kubernetes.io/projected/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2-kube-api-access-9bpsf\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:48:13.424840 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.424804    4931 generic.go:356] "Generic (PLEG): container finished" podID="def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" containerID="e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7" exitCode=0
Nov 01 09:48:13.425319 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.424863    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"
Nov 01 09:48:13.425319 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.424890    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x" event={"ID":"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2","Type":"ContainerDied","Data":"e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7"}
Nov 01 09:48:13.425319 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.424958    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x" event={"ID":"def75dd7-bd4b-4b93-a463-a4fbbbeb85a2","Type":"ContainerDied","Data":"543ae32af27ae36a57fc9bd3a4206b2d0f31dab0773b2cb4c82731df8d0fcdd6"}
Nov 01 09:48:13.425319 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.424983    4931 scope.go:122] "RemoveContainer" containerID="e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7"
Nov 01 09:48:13.436597 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.436574    4931 scope.go:122] "RemoveContainer" containerID="e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7"
Nov 01 09:48:13.436879 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:48:13.436860    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7\": container with ID starting with e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7 not found: ID does not exist" containerID="e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7"
Nov 01 09:48:13.436936 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.436890    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7"} err="failed to get container status \"e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7\": rpc error: code = NotFound desc = could not find container \"e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7\": container with ID starting with e765b175b3061d53cf0d7e6d596a2dc1539c0cb17f68cf95c48d39d105678ca7 not found: ID does not exist"
Nov 01 09:48:13.588646 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.588611    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"]
Nov 01 09:48:13.634896 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:13.634856    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-12a01a-595dd9d4d8-w7n4x"]
Nov 01 09:48:14.049557 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:48:14.049526    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" path="/var/lib/kubelet/pods/def75dd7-bd4b-4b93-a463-a4fbbbeb85a2/volumes"
Nov 01 09:48:31.023007 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:48:31.022966    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:49:01.023082 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:49:01.023017    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:49:13.069361 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.069326    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"]
Nov 01 09:49:13.069870 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.069573    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2" podUID="55b1b828-311a-4e06-891f-0daeb0fbadd1" containerName="maas-api" containerID="cri-o://6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d" gracePeriod=30
Nov 01 09:49:13.451880 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.451845    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:49:13.601532 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.601495    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls\") pod \"55b1b828-311a-4e06-891f-0daeb0fbadd1\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") "
Nov 01 09:49:13.601727 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.601712    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b1b828-311a-4e06-891f-0daeb0fbadd1-kube-api-access-gwwsp\" (UniqueName: \"kubernetes.io/projected/55b1b828-311a-4e06-891f-0daeb0fbadd1-kube-api-access-gwwsp\") pod \"55b1b828-311a-4e06-891f-0daeb0fbadd1\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") "
Nov 01 09:49:13.601801 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.601759    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b1b828-311a-4e06-891f-0daeb0fbadd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b1b828-311a-4e06-891f-0daeb0fbadd1-openshift-service-ca\") pod \"55b1b828-311a-4e06-891f-0daeb0fbadd1\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") "
Nov 01 09:49:13.601801 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.601787    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-metrics-certs\") pod \"55b1b828-311a-4e06-891f-0daeb0fbadd1\" (UID: \"55b1b828-311a-4e06-891f-0daeb0fbadd1\") "
Nov 01 09:49:13.602985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.602951    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b1b828-311a-4e06-891f-0daeb0fbadd1-openshift-service-ca" pod "55b1b828-311a-4e06-891f-0daeb0fbadd1" (UID: "55b1b828-311a-4e06-891f-0daeb0fbadd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:49:13.604195 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.604168    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls" pod "55b1b828-311a-4e06-891f-0daeb0fbadd1" (UID: "55b1b828-311a-4e06-891f-0daeb0fbadd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:49:13.606736 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.606706    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b1b828-311a-4e06-891f-0daeb0fbadd1-kube-api-access-gwwsp" pod "55b1b828-311a-4e06-891f-0daeb0fbadd1" (UID: "55b1b828-311a-4e06-891f-0daeb0fbadd1"). InnerVolumeSpecName "kube-api-access-gwwsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:49:13.607048 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.607008    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-metrics-certs" pod "55b1b828-311a-4e06-891f-0daeb0fbadd1" (UID: "55b1b828-311a-4e06-891f-0daeb0fbadd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:49:13.704212 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.704129    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:13.704212 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.704162    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwwsp\" (UniqueName: \"kubernetes.io/projected/55b1b828-311a-4e06-891f-0daeb0fbadd1-kube-api-access-gwwsp\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:13.704212 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.704180    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b1b828-311a-4e06-891f-0daeb0fbadd1-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:13.704212 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.704194    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b1b828-311a-4e06-891f-0daeb0fbadd1-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:13.839895 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.839858    4931 generic.go:356] "Generic (PLEG): container finished" podID="55b1b828-311a-4e06-891f-0daeb0fbadd1" containerID="6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d" exitCode=0
Nov 01 09:49:13.840097 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.839917    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"
Nov 01 09:49:13.840097 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.839941    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2" event={"ID":"55b1b828-311a-4e06-891f-0daeb0fbadd1","Type":"ContainerDied","Data":"6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d"}
Nov 01 09:49:13.840097 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.840002    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2" event={"ID":"55b1b828-311a-4e06-891f-0daeb0fbadd1","Type":"ContainerDied","Data":"dc8a4ebfa9a944209cb4c275b7fe423b9ddfe09c0d0d54093c5ee8d45c4d0d23"}
Nov 01 09:49:13.840097 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.840059    4931 scope.go:122] "RemoveContainer" containerID="6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d"
Nov 01 09:49:13.850661 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.850641    4931 scope.go:122] "RemoveContainer" containerID="6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d"
Nov 01 09:49:13.851011 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:49:13.850961    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d\": container with ID starting with 6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d not found: ID does not exist" containerID="6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d"
Nov 01 09:49:13.851106 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:13.851019    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d"} err="failed to get container status \"6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d\": rpc error: code = NotFound desc = could not find container \"6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d\": container with ID starting with 6aa33bee78aed12eb70819a805552937a457b55d5aaea15a6b94415e5298d74d not found: ID does not exist"
Nov 01 09:49:14.009175 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:14.009081    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"]
Nov 01 09:49:14.013351 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:14.013318    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3d6bb-744b599c98-t25w2"]
Nov 01 09:49:14.083540 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:14.083495    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b1b828-311a-4e06-891f-0daeb0fbadd1" path="/var/lib/kubelet/pods/55b1b828-311a-4e06-891f-0daeb0fbadd1/volumes"
Nov 01 09:49:31.026804 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:49:31.024693    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:49:33.981251 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:33.981212    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"]
Nov 01 09:49:33.981889 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:33.981445    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p" podUID="4b639a78-6674-4a33-8e9a-56f0e84e70c0" containerName="maas-api" containerID="cri-o://47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f" gracePeriod=30
Nov 01 09:49:34.329942 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.329914    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:49:34.386807 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.386775    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b639a78-6674-4a33-8e9a-56f0e84e70c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b639a78-6674-4a33-8e9a-56f0e84e70c0-openshift-service-ca\") pod \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") "
Nov 01 09:49:34.387060 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.387014    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b639a78-6674-4a33-8e9a-56f0e84e70c0-kube-api-access-clq86\" (UniqueName: \"kubernetes.io/projected/4b639a78-6674-4a33-8e9a-56f0e84e70c0-kube-api-access-clq86\") pod \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") "
Nov 01 09:49:34.387157 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.387107    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls\") pod \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") "
Nov 01 09:49:34.387224 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.387193    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-metrics-certs\") pod \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\" (UID: \"4b639a78-6674-4a33-8e9a-56f0e84e70c0\") "
Nov 01 09:49:34.388593 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.388560    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b639a78-6674-4a33-8e9a-56f0e84e70c0-openshift-service-ca" pod "4b639a78-6674-4a33-8e9a-56f0e84e70c0" (UID: "4b639a78-6674-4a33-8e9a-56f0e84e70c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:49:34.411666 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.411627    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b639a78-6674-4a33-8e9a-56f0e84e70c0-kube-api-access-clq86" pod "4b639a78-6674-4a33-8e9a-56f0e84e70c0" (UID: "4b639a78-6674-4a33-8e9a-56f0e84e70c0"). InnerVolumeSpecName "kube-api-access-clq86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:49:34.415360 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.415254    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls" pod "4b639a78-6674-4a33-8e9a-56f0e84e70c0" (UID: "4b639a78-6674-4a33-8e9a-56f0e84e70c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:49:34.415360 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.415333    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-metrics-certs" pod "4b639a78-6674-4a33-8e9a-56f0e84e70c0" (UID: "4b639a78-6674-4a33-8e9a-56f0e84e70c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:49:34.488754 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.488714    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b639a78-6674-4a33-8e9a-56f0e84e70c0-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:34.488754 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.488746    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clq86\" (UniqueName: \"kubernetes.io/projected/4b639a78-6674-4a33-8e9a-56f0e84e70c0-kube-api-access-clq86\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:34.488754 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.488758    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:34.489095 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.488767    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b639a78-6674-4a33-8e9a-56f0e84e70c0-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:49:34.990856 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.990822    4931 generic.go:356] "Generic (PLEG): container finished" podID="4b639a78-6674-4a33-8e9a-56f0e84e70c0" containerID="47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f" exitCode=0
Nov 01 09:49:34.991407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.991384    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p" event={"ID":"4b639a78-6674-4a33-8e9a-56f0e84e70c0","Type":"ContainerDied","Data":"47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f"}
Nov 01 09:49:34.991530 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.991517    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p" event={"ID":"4b639a78-6674-4a33-8e9a-56f0e84e70c0","Type":"ContainerDied","Data":"2996ef73502c5524fc32c72f09e01f5503ee4cb9ddc0fa04bc7c196cf138ddf8"}
Nov 01 09:49:34.991637 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.991625    4931 scope.go:122] "RemoveContainer" containerID="47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f"
Nov 01 09:49:34.991866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:34.991851    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"
Nov 01 09:49:35.014333 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:35.014307    4931 scope.go:122] "RemoveContainer" containerID="47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f"
Nov 01 09:49:35.014689 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:49:35.014668    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f\": container with ID starting with 47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f not found: ID does not exist" containerID="47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f"
Nov 01 09:49:35.014782 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:35.014697    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f"} err="failed to get container status \"47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f\": rpc error: code = NotFound desc = could not find container \"47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f\": container with ID starting with 47347659edd482fab26614e8103f1c9a398e81aaa3463c23c3a69d480ca0d00f not found: ID does not exist"
Nov 01 09:49:35.151408 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:35.151371    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"]
Nov 01 09:49:35.199545 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:35.199515    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3710de-757b7558bd-d8g7p"]
Nov 01 09:49:36.047936 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:49:36.047904    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b639a78-6674-4a33-8e9a-56f0e84e70c0" path="/var/lib/kubelet/pods/4b639a78-6674-4a33-8e9a-56f0e84e70c0/volumes"
Nov 01 09:50:01.024867 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:50:01.024823    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:50:31.023163 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:50:31.023124    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:51:01.023060 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:51:01.022950    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:51:31.023875 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:51:31.023830    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:52:01.023189 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:52:01.023146    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:52:10.885205 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:10.885174    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:52:10.889244 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:10.889222    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:52:10.909701 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:10.909679    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:52:10.914133 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:10.914111    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:52:31.023999 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:52:31.023910    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:52:43.208845 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.208812    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"]
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209322    4931 cpu_manager.go:414] "containerMap: removing container" podUID="310768c0-2031-445b-8014-d7c4ecef7fb4" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209334    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310768c0-2031-445b-8014-d7c4ecef7fb4" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209349    4931 cpu_manager.go:414] "containerMap: removing container" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209355    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b11e6f0-b4ef-41c0-a5c6-57319ae3f7b0" containerName="cleanup"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209364    4931 cpu_manager.go:414] "containerMap: removing container" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209368    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209389    4931 cpu_manager.go:414] "containerMap: removing container" podUID="4b639a78-6674-4a33-8e9a-56f0e84e70c0" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209395    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b639a78-6674-4a33-8e9a-56f0e84e70c0" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209403    4931 cpu_manager.go:414] "containerMap: removing container" podUID="def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209407    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209414    4931 cpu_manager.go:414] "containerMap: removing container" podUID="4753eca6-0326-4eaa-b4ca-63e2a5caafb4" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209418    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4753eca6-0326-4eaa-b4ca-63e2a5caafb4" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209425    4931 cpu_manager.go:414] "containerMap: removing container" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209429    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209444    4931 cpu_manager.go:414] "containerMap: removing container" podUID="55b1b828-311a-4e06-891f-0daeb0fbadd1" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209450    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b1b828-311a-4e06-891f-0daeb0fbadd1" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209461    4931 cpu_manager.go:414] "containerMap: removing container" podUID="93d9f2ae-4845-4ba9-82b1-9f926b054e35" containerName="maas-api"
Nov 01 09:52:43.209514 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209465    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d9f2ae-4845-4ba9-82b1-9f926b054e35" containerName="maas-api"
Nov 01 09:52:43.210433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209538    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d9f2ae-4845-4ba9-82b1-9f926b054e35" containerName="maas-api"
Nov 01 09:52:43.210433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209547    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="310768c0-2031-445b-8014-d7c4ecef7fb4" containerName="maas-api"
Nov 01 09:52:43.210433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209553    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b838b105-38c3-4633-ad29-98194114fa0c" containerName="cleanup"
Nov 01 09:52:43.210433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209562    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b639a78-6674-4a33-8e9a-56f0e84e70c0" containerName="maas-api"
Nov 01 09:52:43.210433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209568    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="def75dd7-bd4b-4b93-a463-a4fbbbeb85a2" containerName="maas-api"
Nov 01 09:52:43.210433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209573    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4753eca6-0326-4eaa-b4ca-63e2a5caafb4" containerName="maas-api"
Nov 01 09:52:43.210433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209579    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e55226-5dc0-4de0-bd93-2a9236c6aa85" containerName="cleanup"
Nov 01 09:52:43.210433 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.209585    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b1b828-311a-4e06-891f-0daeb0fbadd1" containerName="maas-api"
Nov 01 09:52:43.211880 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.211861    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.264151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.264117    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fd5a10\"/\"openshift-service-ca.crt\""
Nov 01 09:52:43.264151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.264132    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fd5a10\"/\"kube-root-ca.crt\""
Nov 01 09:52:43.311902 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.311868    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fd5a10\"/\"rate-test-model-fd5a10-kserve-self-signed-certs\""
Nov 01 09:52:43.311902 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.311905    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fd5a10\"/\"default-dockercfg-q8jh9\""
Nov 01 09:52:43.315318 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.315293    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"]
Nov 01 09:52:43.322546 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.322523    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-tmp-dir\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.322651 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.322553    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6448b3de-e062-4c1b-be39-1b0871eb4585-tls-certs\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.322651 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.322594    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-home\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.322750 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.322656    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-model-cache\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.322750 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.322687    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-dshm\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.322856 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.322840    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch96z\" (UniqueName: \"kubernetes.io/projected/6448b3de-e062-4c1b-be39-1b0871eb4585-kube-api-access-ch96z\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.423753 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.423712    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-tmp-dir\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.423753 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.423750    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6448b3de-e062-4c1b-be39-1b0871eb4585-tls-certs\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.424003 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.423792    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-home\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.424003 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.423817    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-model-cache\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.424003 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.423856    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-dshm\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.424003 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.423974    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch96z\" (UniqueName: \"kubernetes.io/projected/6448b3de-e062-4c1b-be39-1b0871eb4585-kube-api-access-ch96z\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.424304 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.424281    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-home\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.424371 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.424299    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-tmp-dir\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.424471 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.424449    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-model-cache\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.426481 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.426454    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-dshm\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.426781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.426759    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6448b3de-e062-4c1b-be39-1b0871eb4585-tls-certs\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.486501 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.486427    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch96z\" (UniqueName: \"kubernetes.io/projected/6448b3de-e062-4c1b-be39-1b0871eb4585-kube-api-access-ch96z\") pod \"rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.523538 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.523499    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:43.913346 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:43.913315    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"]
Nov 01 09:52:43.913598 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:52:43.913571    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6448b3de_e062_4c1b_be39_1b0871eb4585.slice/crio-9c479fb27ab68d0e2ae2425aec2bf33345b440344cfeaece4adfe7f5a1e4b4ac WatchSource:0}: Error finding container 9c479fb27ab68d0e2ae2425aec2bf33345b440344cfeaece4adfe7f5a1e4b4ac: Status 404 returned error can't find the container with id 9c479fb27ab68d0e2ae2425aec2bf33345b440344cfeaece4adfe7f5a1e4b4ac
Nov 01 09:52:44.409359 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:44.409316    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6" event={"ID":"6448b3de-e062-4c1b-be39-1b0871eb4585","Type":"ContainerStarted","Data":"2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906"}
Nov 01 09:52:44.409943 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:44.409374    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6" event={"ID":"6448b3de-e062-4c1b-be39-1b0871eb4585","Type":"ContainerStarted","Data":"9c479fb27ab68d0e2ae2425aec2bf33345b440344cfeaece4adfe7f5a1e4b4ac"}
Nov 01 09:52:44.523727 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:44.523693    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:44.533337 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:44.533217    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6" podStartSLOduration=1.5331941100000002 podStartE2EDuration="1.53319411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:52:44.529627442 +0000 UTC m=+4834.644500413" watchObservedRunningTime="2026-11-01 09:52:44.53319411 +0000 UTC m=+4834.648067079"
Nov 01 09:52:53.523709 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:53.523669    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:53.537161 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:53.537134    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:52:54.493151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:52:54.493121    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:53:01.022974 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:53:01.022940    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:53:26.386349 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.386308    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"]
Nov 01 09:53:26.386930 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.386531    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md" podUID="cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" containerName="maas-api" containerID="cri-o://67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b" gracePeriod=30
Nov 01 09:53:26.686429 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.686397    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:53:26.697823 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.697788    4931 generic.go:356] "Generic (PLEG): container finished" podID="cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" containerID="67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b" exitCode=0
Nov 01 09:53:26.697999 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.697828    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"
Nov 01 09:53:26.697999 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.697865    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md" event={"ID":"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0","Type":"ContainerDied","Data":"67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b"}
Nov 01 09:53:26.697999 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.697932    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md" event={"ID":"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0","Type":"ContainerDied","Data":"4433a0426a104a6a390c0654ca5ed89d9ccc8cf2b306f7f7c2cd0cf9d6861e4f"}
Nov 01 09:53:26.697999 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.697949    4931 scope.go:122] "RemoveContainer" containerID="67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b"
Nov 01 09:53:26.710801 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.710776    4931 scope.go:122] "RemoveContainer" containerID="67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b"
Nov 01 09:53:26.711218 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:53:26.711172    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b\": container with ID starting with 67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b not found: ID does not exist" containerID="67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b"
Nov 01 09:53:26.711356 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.711227    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b"} err="failed to get container status \"67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b\": rpc error: code = NotFound desc = could not find container \"67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b\": container with ID starting with 67c83a1f83c7c5fbd4e5297e9878883aabf1d4336c7dc1af698af0b3979a4f7b not found: ID does not exist"
Nov 01 09:53:26.749291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.749251    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-openshift-service-ca\") pod \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") "
Nov 01 09:53:26.749491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.749377    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls\") pod \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") "
Nov 01 09:53:26.749491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.749459    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-kube-api-access-jqk8s\" (UniqueName: \"kubernetes.io/projected/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-kube-api-access-jqk8s\") pod \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") "
Nov 01 09:53:26.749491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.749478    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-metrics-certs\") pod \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\" (UID: \"cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0\") "
Nov 01 09:53:26.749733 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.749700    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-openshift-service-ca" pod "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" (UID: "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:53:26.749895 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.749874    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:26.752524 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.752383    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-kube-api-access-jqk8s" pod "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" (UID: "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0"). InnerVolumeSpecName "kube-api-access-jqk8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:53:26.752524 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.752456    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-metrics-certs" pod "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" (UID: "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:53:26.772315 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.772269    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls" pod "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" (UID: "cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:53:26.852358 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.852316    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqk8s\" (UniqueName: \"kubernetes.io/projected/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-kube-api-access-jqk8s\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:26.852358 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.852360    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:26.852611 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:26.852378    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:53:27.124387 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:27.124353    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"]
Nov 01 09:53:27.127221 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:27.127196    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-035625-6957b4c44d-xr8md"]
Nov 01 09:53:28.048683 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:53:28.048641    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" path="/var/lib/kubelet/pods/cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0/volumes"
Nov 01 09:53:31.023631 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:53:31.023591    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:54:00.978531 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:00.978443    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"]
Nov 01 09:54:01.027107 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:01.027071    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"]
Nov 01 09:54:01.027510 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:54:01.027472    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:54:01.030378 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:01.030339    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"]
Nov 01 09:54:03.628589 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.628558    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"]
Nov 01 09:54:03.629101 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.629084    4931 cpu_manager.go:414] "containerMap: removing container" podUID="cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" containerName="maas-api"
Nov 01 09:54:03.629152 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.629102    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" containerName="maas-api"
Nov 01 09:54:03.629226 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.629217    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfbfa2ef-f8d6-4b93-8f35-fa120348e6e0" containerName="maas-api"
Nov 01 09:54:03.632786 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.632764    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.724403 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.724358    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-home\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.724592 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.724523    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkn4d\" (UniqueName: \"kubernetes.io/projected/9c142cca-f4af-4f69-823d-43286232c40b-kube-api-access-vkn4d\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.724592 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.724585    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-dshm\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.724728 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.724701    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c142cca-f4af-4f69-823d-43286232c40b-tls-certs\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.724815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.724801    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-model-cache\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.724885 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.724865    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-tmp-dir\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.731749 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.731712    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf99\"/\"kube-root-ca.crt\""
Nov 01 09:54:03.731955 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.731811    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf99\"/\"default-dockercfg-t5cr5\""
Nov 01 09:54:03.731955 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.731918    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf99\"/\"rate-test-model-46bf99-kserve-self-signed-certs\""
Nov 01 09:54:03.732227 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.732156    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf99\"/\"openshift-service-ca.crt\""
Nov 01 09:54:03.736342 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.736320    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"]
Nov 01 09:54:03.826701 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.826659    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c142cca-f4af-4f69-823d-43286232c40b-tls-certs\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.826908 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.826770    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-model-cache\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.826908 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.826826    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-tmp-dir\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.827052 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.826921    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-home\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.827115 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.827015    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkn4d\" (UniqueName: \"kubernetes.io/projected/9c142cca-f4af-4f69-823d-43286232c40b-kube-api-access-vkn4d\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.827115 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.827102    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-dshm\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.827382 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.827358    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-model-cache\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.827461 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.827385    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-home\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.827623 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.827604    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-tmp-dir\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.829985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.829956    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-dshm\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.830286 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.830265    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c142cca-f4af-4f69-823d-43286232c40b-tls-certs\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.901628 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.901551    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkn4d\" (UniqueName: \"kubernetes.io/projected/9c142cca-f4af-4f69-823d-43286232c40b-kube-api-access-vkn4d\") pod \"rate-test-model-46bf99-kserve-b676747f7-zdh4t\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") " pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:03.945558 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:03.945522    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:04.336859 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:04.336831    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"]
Nov 01 09:54:04.337796 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:54:04.337772    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c142cca_f4af_4f69_823d_43286232c40b.slice/crio-5746dc49110488b3c55d59c5b6d40c9925d01750e06ab177a71e44f7223ac6ba WatchSource:0}: Error finding container 5746dc49110488b3c55d59c5b6d40c9925d01750e06ab177a71e44f7223ac6ba: Status 404 returned error can't find the container with id 5746dc49110488b3c55d59c5b6d40c9925d01750e06ab177a71e44f7223ac6ba
Nov 01 09:54:04.983352 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:04.983318    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t" event={"ID":"9c142cca-f4af-4f69-823d-43286232c40b","Type":"ContainerStarted","Data":"63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085"}
Nov 01 09:54:04.983882 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:04.983364    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t" event={"ID":"9c142cca-f4af-4f69-823d-43286232c40b","Type":"ContainerStarted","Data":"5746dc49110488b3c55d59c5b6d40c9925d01750e06ab177a71e44f7223ac6ba"}
Nov 01 09:54:05.118828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:05.118774    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t" podStartSLOduration=2.118760221 podStartE2EDuration="2.118760221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:54:05.116802378 +0000 UTC m=+4915.231675348" watchObservedRunningTime="2026-11-01 09:54:05.118760221 +0000 UTC m=+4915.233633189"
Nov 01 09:54:05.945659 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:05.945604    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:13.946400 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:13.946355    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:13.959259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:13.959233    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:14.056994 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:14.056955    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:54:16.011259 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.011220    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main" containerID="cri-o://a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493" gracePeriod=45
Nov 01 09:54:16.051333 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.051290    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main" containerID="cri-o://0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43" gracePeriod=45
Nov 01 09:54:16.056662 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.056492    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main" containerID="cri-o://65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3" gracePeriod=45
Nov 01 09:54:16.385062 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.382041    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 09:54:16.388766 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.388742    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 09:54:16.404700 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.404676    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 09:54:16.414758 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.414735    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-home\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-home\") pod \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") "
Nov 01 09:54:16.414899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.414776    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-tmp-dir\") pod \"1fd69395-69b6-443c-8023-9a83841d9aa6\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") "
Nov 01 09:54:16.414899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.414829    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-dshm\") pod \"1fd69395-69b6-443c-8023-9a83841d9aa6\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") "
Nov 01 09:54:16.414899 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.414867    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tls-certs\" (UniqueName: \"kubernetes.io/secret/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tls-certs\") pod \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") "
Nov 01 09:54:16.415109 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.414912    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-home\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-home\") pod \"1fd69395-69b6-443c-8023-9a83841d9aa6\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") "
Nov 01 09:54:16.415109 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.414946    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-kube-api-access-8mnf4\" (UniqueName: \"kubernetes.io/projected/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-kube-api-access-8mnf4\") pod \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") "
Nov 01 09:54:16.415109 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.414988    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-dshm\") pod \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") "
Nov 01 09:54:16.415109 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415085    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-dshm\") pod \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") "
Nov 01 09:54:16.415323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415166    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tmp-dir\") pod \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") "
Nov 01 09:54:16.415323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415213    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-model-cache\") pod \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") "
Nov 01 09:54:16.415323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415263    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-kube-api-access-vqkjd\" (UniqueName: \"kubernetes.io/projected/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-kube-api-access-vqkjd\") pod \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") "
Nov 01 09:54:16.415491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415332    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd69395-69b6-443c-8023-9a83841d9aa6-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd69395-69b6-443c-8023-9a83841d9aa6-tls-certs\") pod \"1fd69395-69b6-443c-8023-9a83841d9aa6\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") "
Nov 01 09:54:16.415491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415364    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd69395-69b6-443c-8023-9a83841d9aa6-kube-api-access-q9j8v\" (UniqueName: \"kubernetes.io/projected/1fd69395-69b6-443c-8023-9a83841d9aa6-kube-api-access-q9j8v\") pod \"1fd69395-69b6-443c-8023-9a83841d9aa6\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") "
Nov 01 09:54:16.415491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415412    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-model-cache\") pod \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\" (UID: \"dac89622-b6dc-47b0-9b5b-e723e7b4fb55\") "
Nov 01 09:54:16.415491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415441    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tls-certs\" (UniqueName: \"kubernetes.io/secret/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tls-certs\") pod \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") "
Nov 01 09:54:16.415491 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415468    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tmp-dir\") pod \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") "
Nov 01 09:54:16.415748 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415512    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-home\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-home\") pod \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\" (UID: \"5496b80d-d6ec-4409-8b8e-eea80d5b1d01\") "
Nov 01 09:54:16.415748 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.415561    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-model-cache\") pod \"1fd69395-69b6-443c-8023-9a83841d9aa6\" (UID: \"1fd69395-69b6-443c-8023-9a83841d9aa6\") "
Nov 01 09:54:16.416357 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.416332    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-model-cache" pod "1fd69395-69b6-443c-8023-9a83841d9aa6" (UID: "1fd69395-69b6-443c-8023-9a83841d9aa6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.416605 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.416544    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-model-cache" pod "dac89622-b6dc-47b0-9b5b-e723e7b4fb55" (UID: "dac89622-b6dc-47b0-9b5b-e723e7b4fb55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.425793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.419344    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-home" pod "1fd69395-69b6-443c-8023-9a83841d9aa6" (UID: "1fd69395-69b6-443c-8023-9a83841d9aa6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.425793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.423861    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tls-certs" pod "5496b80d-d6ec-4409-8b8e-eea80d5b1d01" (UID: "5496b80d-d6ec-4409-8b8e-eea80d5b1d01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:54:16.425793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.423876    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-dshm" pod "1fd69395-69b6-443c-8023-9a83841d9aa6" (UID: "1fd69395-69b6-443c-8023-9a83841d9aa6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.425793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.424088    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tmp-dir" pod "5496b80d-d6ec-4409-8b8e-eea80d5b1d01" (UID: "5496b80d-d6ec-4409-8b8e-eea80d5b1d01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.425793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.424128    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-home" pod "dac89622-b6dc-47b0-9b5b-e723e7b4fb55" (UID: "dac89622-b6dc-47b0-9b5b-e723e7b4fb55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.425793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.424256    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-home" pod "5496b80d-d6ec-4409-8b8e-eea80d5b1d01" (UID: "5496b80d-d6ec-4409-8b8e-eea80d5b1d01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.425793 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.425725    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tls-certs" pod "dac89622-b6dc-47b0-9b5b-e723e7b4fb55" (UID: "dac89622-b6dc-47b0-9b5b-e723e7b4fb55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:54:16.426776 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.425850    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-tmp-dir" pod "1fd69395-69b6-443c-8023-9a83841d9aa6" (UID: "1fd69395-69b6-443c-8023-9a83841d9aa6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.426776 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.425863    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-model-cache" pod "5496b80d-d6ec-4409-8b8e-eea80d5b1d01" (UID: "5496b80d-d6ec-4409-8b8e-eea80d5b1d01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.426776 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.426106    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd69395-69b6-443c-8023-9a83841d9aa6-kube-api-access-q9j8v" pod "1fd69395-69b6-443c-8023-9a83841d9aa6" (UID: "1fd69395-69b6-443c-8023-9a83841d9aa6"). InnerVolumeSpecName "kube-api-access-q9j8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:16.426776 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.426379    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tmp-dir" pod "dac89622-b6dc-47b0-9b5b-e723e7b4fb55" (UID: "dac89622-b6dc-47b0-9b5b-e723e7b4fb55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.428169 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.428139    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-dshm" pod "dac89622-b6dc-47b0-9b5b-e723e7b4fb55" (UID: "dac89622-b6dc-47b0-9b5b-e723e7b4fb55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.428599 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.428569    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd69395-69b6-443c-8023-9a83841d9aa6-tls-certs" pod "1fd69395-69b6-443c-8023-9a83841d9aa6" (UID: "1fd69395-69b6-443c-8023-9a83841d9aa6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:54:16.428982 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.428955    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-dshm" pod "5496b80d-d6ec-4409-8b8e-eea80d5b1d01" (UID: "5496b80d-d6ec-4409-8b8e-eea80d5b1d01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:54:16.429554 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.429531    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-kube-api-access-8mnf4" pod "dac89622-b6dc-47b0-9b5b-e723e7b4fb55" (UID: "dac89622-b6dc-47b0-9b5b-e723e7b4fb55"). InnerVolumeSpecName "kube-api-access-8mnf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:16.430632 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.430603    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-kube-api-access-vqkjd" pod "5496b80d-d6ec-4409-8b8e-eea80d5b1d01" (UID: "5496b80d-d6ec-4409-8b8e-eea80d5b1d01"). InnerVolumeSpecName "kube-api-access-vqkjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:54:16.517321 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517283    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517321 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517320    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517337    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517350    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517364    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517379    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mnf4\" (UniqueName: \"kubernetes.io/projected/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-kube-api-access-8mnf4\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517397    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517409    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517421    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517434    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517448    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqkjd\" (UniqueName: \"kubernetes.io/projected/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-kube-api-access-vqkjd\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517462    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd69395-69b6-443c-8023-9a83841d9aa6-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517475    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9j8v\" (UniqueName: \"kubernetes.io/projected/1fd69395-69b6-443c-8023-9a83841d9aa6-kube-api-access-q9j8v\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517490    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac89622-b6dc-47b0-9b5b-e723e7b4fb55-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517503    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517517    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517532    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5496b80d-d6ec-4409-8b8e-eea80d5b1d01-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:16.517567 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:16.517544    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd69395-69b6-443c-8023-9a83841d9aa6-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:54:17.078913 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.078879    4931 generic.go:356] "Generic (PLEG): container finished" podID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerID="0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43" exitCode=0
Nov 01 09:54:17.079422 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.078937    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"
Nov 01 09:54:17.079422 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.078960    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" event={"ID":"dac89622-b6dc-47b0-9b5b-e723e7b4fb55","Type":"ContainerDied","Data":"0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43"}
Nov 01 09:54:17.079422 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.079007    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2" event={"ID":"dac89622-b6dc-47b0-9b5b-e723e7b4fb55","Type":"ContainerDied","Data":"5796ceadf9da047ca20a8024b1a0caca7630ba442409981358c086dcc721a2f3"}
Nov 01 09:54:17.079422 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.079056    4931 scope.go:122] "RemoveContainer" containerID="0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43"
Nov 01 09:54:17.080828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.080641    4931 generic.go:356] "Generic (PLEG): container finished" podID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerID="65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3" exitCode=0
Nov 01 09:54:17.080828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.080732    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"
Nov 01 09:54:17.080828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.080736    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" event={"ID":"1fd69395-69b6-443c-8023-9a83841d9aa6","Type":"ContainerDied","Data":"65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3"}
Nov 01 09:54:17.080828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.080780    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j" event={"ID":"1fd69395-69b6-443c-8023-9a83841d9aa6","Type":"ContainerDied","Data":"35c84f0000076410ea64d1464ccdaf5d093c6fcba47a3ad7af3ed1b895ee1da9"}
Nov 01 09:54:17.082577 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.082554    4931 generic.go:356] "Generic (PLEG): container finished" podID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerID="a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493" exitCode=0
Nov 01 09:54:17.082708 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.082585    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" event={"ID":"5496b80d-d6ec-4409-8b8e-eea80d5b1d01","Type":"ContainerDied","Data":"a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493"}
Nov 01 09:54:17.082708 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.082616    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"
Nov 01 09:54:17.082708 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.082619    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8" event={"ID":"5496b80d-d6ec-4409-8b8e-eea80d5b1d01","Type":"ContainerDied","Data":"ea9b78b60274e733271291b13f2eaa2463ffcd922e18cfd585f953763b44dd96"}
Nov 01 09:54:17.090177 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.089976    4931 scope.go:122] "RemoveContainer" containerID="0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43"
Nov 01 09:54:17.090351 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:54:17.090329    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43\": container with ID starting with 0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43 not found: ID does not exist" containerID="0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43"
Nov 01 09:54:17.090419 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.090360    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43"} err="failed to get container status \"0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43\": rpc error: code = NotFound desc = could not find container \"0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43\": container with ID starting with 0b9ca1fe300c32162e31522035f4ec6803a66c3de39d6940f2aac70264743b43 not found: ID does not exist"
Nov 01 09:54:17.090419 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.090379    4931 scope.go:122] "RemoveContainer" containerID="65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3"
Nov 01 09:54:17.101487 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.101472    4931 scope.go:122] "RemoveContainer" containerID="65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3"
Nov 01 09:54:17.101751 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:54:17.101730    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3\": container with ID starting with 65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3 not found: ID does not exist" containerID="65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3"
Nov 01 09:54:17.101844 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.101755    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3"} err="failed to get container status \"65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3\": rpc error: code = NotFound desc = could not find container \"65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3\": container with ID starting with 65918bd58e67a77e26eb8ef020dbfafa63995c6d21339fcba9be92b8bbc7b4c3 not found: ID does not exist"
Nov 01 09:54:17.101844 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.101771    4931 scope.go:122] "RemoveContainer" containerID="a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493"
Nov 01 09:54:17.110503 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.110490    4931 scope.go:122] "RemoveContainer" containerID="a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493"
Nov 01 09:54:17.110760 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:54:17.110739    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493\": container with ID starting with a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493 not found: ID does not exist" containerID="a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493"
Nov 01 09:54:17.110846 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.110765    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493"} err="failed to get container status \"a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493\": rpc error: code = NotFound desc = could not find container \"a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493\": container with ID starting with a2d5cfe52252e01559efffe3c6604f65519d5b6a38beed4e9af3da90a0a1a493 not found: ID does not exist"
Nov 01 09:54:17.250550 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.250518    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"]
Nov 01 09:54:17.297847 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.297769    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/distinct-2-w3-54e01b-kserve-858467db6b-9mzl8"]
Nov 01 09:54:17.419049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.416983    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"]
Nov 01 09:54:17.432048 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.421959    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/premium-w3-54e01b-kserve-fdfb9dd8b-spgf2"]
Nov 01 09:54:17.518448 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.518419    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"]
Nov 01 09:54:17.525084 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:17.525041    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/simulator-w3-54e01b-kserve-d5c956f75-qq55j"]
Nov 01 09:54:18.047549 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:18.047516    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" path="/var/lib/kubelet/pods/1fd69395-69b6-443c-8023-9a83841d9aa6/volumes"
Nov 01 09:54:18.047936 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:18.047923    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" path="/var/lib/kubelet/pods/5496b80d-d6ec-4409-8b8e-eea80d5b1d01/volumes"
Nov 01 09:54:18.048320 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:54:18.048307    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" path="/var/lib/kubelet/pods/dac89622-b6dc-47b0-9b5b-e723e7b4fb55/volumes"
Nov 01 09:54:31.023302 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:54:31.023262    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:55:01.026323 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:55:01.026285    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:55:06.405948 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.405895    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"]
Nov 01 09:55:06.406521 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.406174    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl" podUID="e6ebc602-27d4-4f59-b8d0-f5a99d58b973" containerName="maas-api" containerID="cri-o://22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0" gracePeriod=30
Nov 01 09:55:06.724509 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.724479    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 09:55:06.823067 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.819608    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-metrics-certs\") pod \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") "
Nov 01 09:55:06.823067 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.819681    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-kube-api-access-k5jd4\" (UniqueName: \"kubernetes.io/projected/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-kube-api-access-k5jd4\") pod \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") "
Nov 01 09:55:06.823067 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.819750    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-openshift-service-ca\") pod \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") "
Nov 01 09:55:06.823067 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.819917    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls\") pod \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\" (UID: \"e6ebc602-27d4-4f59-b8d0-f5a99d58b973\") "
Nov 01 09:55:06.824230 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.824174    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-openshift-service-ca" pod "e6ebc602-27d4-4f59-b8d0-f5a99d58b973" (UID: "e6ebc602-27d4-4f59-b8d0-f5a99d58b973"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:55:06.840942 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.840835    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-metrics-certs" pod "e6ebc602-27d4-4f59-b8d0-f5a99d58b973" (UID: "e6ebc602-27d4-4f59-b8d0-f5a99d58b973"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:06.842254 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.842217    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls" pod "e6ebc602-27d4-4f59-b8d0-f5a99d58b973" (UID: "e6ebc602-27d4-4f59-b8d0-f5a99d58b973"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:06.845424 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.845328    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-kube-api-access-k5jd4" pod "e6ebc602-27d4-4f59-b8d0-f5a99d58b973" (UID: "e6ebc602-27d4-4f59-b8d0-f5a99d58b973"). InnerVolumeSpecName "kube-api-access-k5jd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:06.921584 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.921547    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:06.921584 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.921581    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5jd4\" (UniqueName: \"kubernetes.io/projected/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-kube-api-access-k5jd4\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:06.921584 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.921591    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:06.921819 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:06.921602    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ebc602-27d4-4f59-b8d0-f5a99d58b973-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:07.440018 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.439971    4931 generic.go:356] "Generic (PLEG): container finished" podID="e6ebc602-27d4-4f59-b8d0-f5a99d58b973" containerID="22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0" exitCode=0
Nov 01 09:55:07.440574 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.440112    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"
Nov 01 09:55:07.440941 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.440674    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl" event={"ID":"e6ebc602-27d4-4f59-b8d0-f5a99d58b973","Type":"ContainerDied","Data":"22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0"}
Nov 01 09:55:07.440941 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.440731    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl" event={"ID":"e6ebc602-27d4-4f59-b8d0-f5a99d58b973","Type":"ContainerDied","Data":"c3a31a66f0c8512e63c219ad2e51ce52b35e828a11716c2a10b99ecddc842dfa"}
Nov 01 09:55:07.440941 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.440757    4931 scope.go:122] "RemoveContainer" containerID="22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0"
Nov 01 09:55:07.456296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.456272    4931 scope.go:122] "RemoveContainer" containerID="22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0"
Nov 01 09:55:07.456594 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:55:07.456575    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0\": container with ID starting with 22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0 not found: ID does not exist" containerID="22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0"
Nov 01 09:55:07.456660 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.456607    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0"} err="failed to get container status \"22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0\": rpc error: code = NotFound desc = could not find container \"22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0\": container with ID starting with 22792551b8578d1fc1361d8522eb2fe95b939b2d18322023c072bd14c10693f0 not found: ID does not exist"
Nov 01 09:55:07.609225 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.609192    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"]
Nov 01 09:55:07.655985 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:07.655949    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-54e01b-86b54b8757-rg2tl"]
Nov 01 09:55:08.048012 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:08.047981    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ebc602-27d4-4f59-b8d0-f5a99d58b973" path="/var/lib/kubelet/pods/e6ebc602-27d4-4f59-b8d0-f5a99d58b973/volumes"
Nov 01 09:55:16.310722 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:16.310687    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"]
Nov 01 09:55:16.804494 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:16.804456    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"]
Nov 01 09:55:16.804749 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:16.804724    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" containerID="cri-o://e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19" gracePeriod=30
Nov 01 09:55:19.185949 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:19.185911    4931 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:55:19.186367 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:19.185969    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:55:22.079384 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.079326    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 09:55:22.197994 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.197961    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-token\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-token\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198221 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198003    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-podinfo\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198221 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198116    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-certs\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198221 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198157    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-socket\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198221 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198183    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-envoy\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198497 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198242    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-data\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198497 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198307    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istiod-ca-cert\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198497 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198340    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-kube-api-access-2tnhd\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-kube-api-access-2tnhd\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198497 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198486    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-credential-socket\") pod \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\" (UID: \"a021d984-a9f1-4c70-b6b5-c4d924d27e99\") "
Nov 01 09:55:22.198694 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198590    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-socket" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:22.198694 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198675    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istiod-ca-cert" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:55:22.198878 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198828    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-certs" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:22.198878 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198858    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-credential-socket" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:22.198878 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.198876    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-data" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:22.199072 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.199045    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.199111 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.199080    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.199111 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.199093    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.199111 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.199107    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.199230 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.199121    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.201091 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.201065    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-podinfo" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:55:22.201214 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.201129    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-token" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:22.201446 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.201427    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-kube-api-access-2tnhd" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "kube-api-access-2tnhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:22.201526 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.201450    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-envoy" pod "a021d984-a9f1-4c70-b6b5-c4d924d27e99" (UID: "a021d984-a9f1-4c70-b6b5-c4d924d27e99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:22.300494 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.300460    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.300494 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.300491    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tnhd\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-kube-api-access-2tnhd\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.300494 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.300504    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.300732 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.300513    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a021d984-a9f1-4c70-b6b5-c4d924d27e99-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:22.549281 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.549251    4931 generic.go:356] "Generic (PLEG): container finished" podID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerID="e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19" exitCode=0
Nov 01 09:55:22.549503 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.549301    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"
Nov 01 09:55:22.549503 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.549341    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" event={"ID":"a021d984-a9f1-4c70-b6b5-c4d924d27e99","Type":"ContainerDied","Data":"e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19"}
Nov 01 09:55:22.549503 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.549408    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z" event={"ID":"a021d984-a9f1-4c70-b6b5-c4d924d27e99","Type":"ContainerDied","Data":"93af4489d3b11532203533160773f51ebd6f0041e252171cbadc0ad5a6f81b2c"}
Nov 01 09:55:22.549503 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.549438    4931 scope.go:122] "RemoveContainer" containerID="e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19"
Nov 01 09:55:22.560323 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.560302    4931 scope.go:122] "RemoveContainer" containerID="e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19"
Nov 01 09:55:22.560617 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:55:22.560596    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19\": container with ID starting with e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19 not found: ID does not exist" containerID="e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19"
Nov 01 09:55:22.560703 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.560630    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19"} err="failed to get container status \"e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19\": rpc error: code = NotFound desc = could not find container \"e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19\": container with ID starting with e32a2e0ced404e13b3ba0c0574b0ad59aeab46c8b9f0143808fe4fda92867c19 not found: ID does not exist"
Nov 01 09:55:22.709577 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.709546    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"]
Nov 01 09:55:22.757500 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:22.757469    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-bcm8z"]
Nov 01 09:55:24.047759 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:24.047727    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" path="/var/lib/kubelet/pods/a021d984-a9f1-4c70-b6b5-c4d924d27e99/volumes"
Nov 01 09:55:31.025624 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:55:31.025580    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:55:31.333137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.333018    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main" containerID="cri-o://161b086b599982286e140c30bc87e8f3740a0b5621d349ae49abca1d7d65138f" gracePeriod=45
Nov 01 09:55:31.612849 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.612811    4931 generic.go:356] "Generic (PLEG): container finished" podID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerID="161b086b599982286e140c30bc87e8f3740a0b5621d349ae49abca1d7d65138f" exitCode=0
Nov 01 09:55:31.613107 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.612867    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" event={"ID":"0d12efcf-78f3-424b-ab14-e0f8f756e930","Type":"ContainerDied","Data":"161b086b599982286e140c30bc87e8f3740a0b5621d349ae49abca1d7d65138f"}
Nov 01 09:55:31.680064 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.678057    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.846478    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d12efcf-78f3-424b-ab14-e0f8f756e930-kube-api-access-4dwkv\" (UniqueName: \"kubernetes.io/projected/0d12efcf-78f3-424b-ab14-e0f8f756e930-kube-api-access-4dwkv\") pod \"0d12efcf-78f3-424b-ab14-e0f8f756e930\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") "
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.846579    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-tmp-dir\") pod \"0d12efcf-78f3-424b-ab14-e0f8f756e930\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") "
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.846645    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d12efcf-78f3-424b-ab14-e0f8f756e930-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d12efcf-78f3-424b-ab14-e0f8f756e930-tls-certs\") pod \"0d12efcf-78f3-424b-ab14-e0f8f756e930\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") "
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.846682    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-dshm\") pod \"0d12efcf-78f3-424b-ab14-e0f8f756e930\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") "
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.846758    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-home\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-home\") pod \"0d12efcf-78f3-424b-ab14-e0f8f756e930\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") "
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.846789    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-model-cache\") pod \"0d12efcf-78f3-424b-ab14-e0f8f756e930\" (UID: \"0d12efcf-78f3-424b-ab14-e0f8f756e930\") "
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.846936    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-tmp-dir" pod "0d12efcf-78f3-424b-ab14-e0f8f756e930" (UID: "0d12efcf-78f3-424b-ab14-e0f8f756e930"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.847291    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.847646    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-model-cache" pod "0d12efcf-78f3-424b-ab14-e0f8f756e930" (UID: "0d12efcf-78f3-424b-ab14-e0f8f756e930"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.849513 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.847659    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-home" pod "0d12efcf-78f3-424b-ab14-e0f8f756e930" (UID: "0d12efcf-78f3-424b-ab14-e0f8f756e930"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.852713 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.852679    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d12efcf-78f3-424b-ab14-e0f8f756e930-kube-api-access-4dwkv" pod "0d12efcf-78f3-424b-ab14-e0f8f756e930" (UID: "0d12efcf-78f3-424b-ab14-e0f8f756e930"). InnerVolumeSpecName "kube-api-access-4dwkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:55:31.876573 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.876515    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d12efcf-78f3-424b-ab14-e0f8f756e930-tls-certs" pod "0d12efcf-78f3-424b-ab14-e0f8f756e930" (UID: "0d12efcf-78f3-424b-ab14-e0f8f756e930"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:55:31.877208 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.877168    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-dshm" pod "0d12efcf-78f3-424b-ab14-e0f8f756e930" (UID: "0d12efcf-78f3-424b-ab14-e0f8f756e930"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:55:31.947905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.947867    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dwkv\" (UniqueName: \"kubernetes.io/projected/0d12efcf-78f3-424b-ab14-e0f8f756e930-kube-api-access-4dwkv\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.947905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.947900    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d12efcf-78f3-424b-ab14-e0f8f756e930-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.947905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.947909    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.948166 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.947919    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:31.948166 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:31.947927    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d12efcf-78f3-424b-ab14-e0f8f756e930-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:55:32.621227 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:32.621188    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j" event={"ID":"0d12efcf-78f3-424b-ab14-e0f8f756e930","Type":"ContainerDied","Data":"b689c871146f304f621a455ecbe0918e0e5f81764c3156d69e9f619380540ea7"}
Nov 01 09:55:32.621702 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:32.621239    4931 scope.go:122] "RemoveContainer" containerID="161b086b599982286e140c30bc87e8f3740a0b5621d349ae49abca1d7d65138f"
Nov 01 09:55:32.621702 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:32.621353    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"
Nov 01 09:55:32.783306 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:32.783270    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"]
Nov 01 09:55:32.830816 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:32.830779    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-343207/distinct-2-w2-343207-kserve-cbd59b487-thv8j"]
Nov 01 09:55:34.048530 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:55:34.048489    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" path="/var/lib/kubelet/pods/0d12efcf-78f3-424b-ab14-e0f8f756e930/volumes"
Nov 01 09:56:01.023326 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:56:01.023283    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:56:11.133884 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:11.133847    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"]
Nov 01 09:56:11.134328 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:11.134155    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy" containerID="cri-o://7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6" gracePeriod=30
Nov 01 09:56:16.394812 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.394789    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 09:56:16.464497 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464408    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8971d8cb-bc36-424c-8523-3b31df38f235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8971d8cb-bc36-424c-8523-3b31df38f235-istio-podinfo\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464497 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464449    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-istio-token\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-istio-token\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464560    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-data\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464606    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8971d8cb-bc36-424c-8523-3b31df38f235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8971d8cb-bc36-424c-8523-3b31df38f235-istiod-ca-cert\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464642    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-socket\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464741 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464689    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-certs\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464995 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464749    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-kube-api-access-b4n5n\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-kube-api-access-b4n5n\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464995 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464792    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-envoy\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464995 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464825    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-credential-socket\") pod \"8971d8cb-bc36-424c-8523-3b31df38f235\" (UID: \"8971d8cb-bc36-424c-8523-3b31df38f235\") "
Nov 01 09:56:16.464995 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464877    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-data" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:16.464995 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464983    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8971d8cb-bc36-424c-8523-3b31df38f235-istiod-ca-cert" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:56:16.465528 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.464999    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-socket" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:16.465528 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.465315    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-certs" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:16.465528 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.465332    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-credential-socket" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:16.465828 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.465807    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.465897 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.465835    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.465897 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.465850    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8971d8cb-bc36-424c-8523-3b31df38f235-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.465897 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.465865    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.465897 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.465881    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.467056 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.467012    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8971d8cb-bc36-424c-8523-3b31df38f235-istio-podinfo" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:56:16.467162 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.467093    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-istio-token" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:56:16.467162 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.467144    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-envoy" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:56:16.467278 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.467262    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-kube-api-access-b4n5n" pod "8971d8cb-bc36-424c-8523-3b31df38f235" (UID: "8971d8cb-bc36-424c-8523-3b31df38f235"). InnerVolumeSpecName "kube-api-access-b4n5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:56:16.567183 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.567148    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4n5n\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-kube-api-access-b4n5n\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.567183 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.567181    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8971d8cb-bc36-424c-8523-3b31df38f235-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.567183 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.567195    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8971d8cb-bc36-424c-8523-3b31df38f235-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.567482 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.567208    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8971d8cb-bc36-424c-8523-3b31df38f235-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:56:16.916522 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.916487    4931 generic.go:356] "Generic (PLEG): container finished" podID="8971d8cb-bc36-424c-8523-3b31df38f235" containerID="7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6" exitCode=0
Nov 01 09:56:16.916726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.916546    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"
Nov 01 09:56:16.916726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.916567    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" event={"ID":"8971d8cb-bc36-424c-8523-3b31df38f235","Type":"ContainerDied","Data":"7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6"}
Nov 01 09:56:16.916726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.916623    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs" event={"ID":"8971d8cb-bc36-424c-8523-3b31df38f235","Type":"ContainerDied","Data":"b5c5a018b5b319678dc64656a3f9b61332f1e07aae94f4bbe0e8fa0bbb4f01f7"}
Nov 01 09:56:16.916726 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.916647    4931 scope.go:122] "RemoveContainer" containerID="7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6"
Nov 01 09:56:16.927525 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.927506    4931 scope.go:122] "RemoveContainer" containerID="7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6"
Nov 01 09:56:16.927783 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:56:16.927766    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6\": container with ID starting with 7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6 not found: ID does not exist" containerID="7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6"
Nov 01 09:56:16.927832 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:16.927792    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6"} err="failed to get container status \"7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6\": rpc error: code = NotFound desc = could not find container \"7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6\": container with ID starting with 7d104bec89b3af24bbcd9a122daf67f85e96e231ff4314d0e2251e30c4d141b6 not found: ID does not exist"
Nov 01 09:56:17.076971 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:17.076938    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"]
Nov 01 09:56:17.081322 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:17.081293    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-p7lbs"]
Nov 01 09:56:18.047485 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:56:18.047456    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" path="/var/lib/kubelet/pods/8971d8cb-bc36-424c-8523-3b31df38f235/volumes"
Nov 01 09:56:31.022450 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:56:31.022407    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:57:01.022063 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:57:01.021966    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:57:10.965706 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:10.965596    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:57:10.970756 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:10.970730    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:57:11.011664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:11.011634    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 09:57:11.018445 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:11.018425    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 09:57:31.027197 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:57:31.027154    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:57:32.659238 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:32.659204    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"]
Nov 01 09:57:32.659849 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:32.659421    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy" containerID="cri-o://55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f" gracePeriod=30
Nov 01 09:57:37.933992 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:37.933966    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:57:38.065258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065173    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-data\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.065258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065222    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-podinfo\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.065258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065255    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-socket\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.065527 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065349    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-envoy\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.065527 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065419    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a1f4a0-3751-45de-9fc7-89188a792f50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a1f4a0-3751-45de-9fc7-89188a792f50-istiod-ca-cert\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.065527 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065447    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-token\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-token\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.065527 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065497    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-kube-api-access-ljs2z\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-kube-api-access-ljs2z\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.065721 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065534    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-certs\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.065721 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.065595    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-credential-socket\") pod \"a8a1f4a0-3751-45de-9fc7-89188a792f50\" (UID: \"a8a1f4a0-3751-45de-9fc7-89188a792f50\") "
Nov 01 09:57:38.066664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.066085    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-data" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:38.066664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.066267    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-certs" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:38.066664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.066401    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.066664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.066421    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.066664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.066464    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-credential-socket" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:38.066664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.066535    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-socket" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:38.066664 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.066588    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a1f4a0-3751-45de-9fc7-89188a792f50-istiod-ca-cert" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:57:38.069326 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.069298    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-kube-api-access-ljs2z" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "kube-api-access-ljs2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:38.071492 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.071449    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-envoy" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:57:38.074062 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.073993    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-podinfo" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:57:38.074541 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.074518    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-token" pod "a8a1f4a0-3751-45de-9fc7-89188a792f50" (UID: "a8a1f4a0-3751-45de-9fc7-89188a792f50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:57:38.167832 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.167793    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a1f4a0-3751-45de-9fc7-89188a792f50-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.167832 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.167829    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.168093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.167844    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljs2z\" (UniqueName: \"kubernetes.io/projected/a8a1f4a0-3751-45de-9fc7-89188a792f50-kube-api-access-ljs2z\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.168093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.167859    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.168093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.167873    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.168093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.167886    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.168093 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.167899    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a1f4a0-3751-45de-9fc7-89188a792f50-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:57:38.462774 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.462735    4931 generic.go:356] "Generic (PLEG): container finished" podID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerID="55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f" exitCode=0
Nov 01 09:57:38.462948 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.462800    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"
Nov 01 09:57:38.462948 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.462831    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" event={"ID":"a8a1f4a0-3751-45de-9fc7-89188a792f50","Type":"ContainerDied","Data":"55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f"}
Nov 01 09:57:38.462948 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.462899    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf" event={"ID":"a8a1f4a0-3751-45de-9fc7-89188a792f50","Type":"ContainerDied","Data":"7d89a3f41de40e1040442f97009bd6a053d308eb3c0ffa27a90fa3b30b8372bb"}
Nov 01 09:57:38.462948 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.462928    4931 scope.go:122] "RemoveContainer" containerID="55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f"
Nov 01 09:57:38.473545 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.473357    4931 scope.go:122] "RemoveContainer" containerID="55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f"
Nov 01 09:57:38.473730 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:57:38.473699    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f\": container with ID starting with 55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f not found: ID does not exist" containerID="55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f"
Nov 01 09:57:38.473818 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.473744    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f"} err="failed to get container status \"55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f\": rpc error: code = NotFound desc = could not find container \"55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f\": container with ID starting with 55dcb933a432a69303cb41487e21d650d6b020fa59dab65cc33ec7ccee9fdd9f not found: ID does not exist"
Nov 01 09:57:38.622544 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.622512    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"]
Nov 01 09:57:38.626540 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:38.626501    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rvlsf"]
Nov 01 09:57:40.047572 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:57:40.047539    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" path="/var/lib/kubelet/pods/a8a1f4a0-3751-45de-9fc7-89188a792f50/volumes"
Nov 01 09:58:01.022229 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:58:01.022185    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:58:07.877411 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:07.877374    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"]
Nov 01 09:58:08.164473 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:08.164389    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"]
Nov 01 09:58:08.170803 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:08.170775    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"]
Nov 01 09:58:08.190575 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:08.190544    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"]
Nov 01 09:58:16.463548 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:16.463499    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"]
Nov 01 09:58:22.897846 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:22.897753    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6" podUID="6448b3de-e062-4c1b-be39-1b0871eb4585" containerName="main" containerID="cri-o://2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906" gracePeriod=45
Nov 01 09:58:23.152550 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.152497    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:58:23.185758 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.185703    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k" podUID="bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" containerName="main" containerID="cri-o://ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab" gracePeriod=45
Nov 01 09:58:23.191147 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.191114    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main" containerID="cri-o://23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef" gracePeriod=45
Nov 01 09:58:23.207271 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.207239    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": dial tcp 10.133.0.88:8000: connect: connection refused"
Nov 01 09:58:23.210001 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.209922    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" containerID="cri-o://d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb" gracePeriod=45
Nov 01 09:58:23.238681 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.238653    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-model-cache\") pod \"6448b3de-e062-4c1b-be39-1b0871eb4585\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") "
Nov 01 09:58:23.238785 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.238689    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-dshm\") pod \"6448b3de-e062-4c1b-be39-1b0871eb4585\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") "
Nov 01 09:58:23.238785 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.238760    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6448b3de-e062-4c1b-be39-1b0871eb4585-kube-api-access-ch96z\" (UniqueName: \"kubernetes.io/projected/6448b3de-e062-4c1b-be39-1b0871eb4585-kube-api-access-ch96z\") pod \"6448b3de-e062-4c1b-be39-1b0871eb4585\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") "
Nov 01 09:58:23.238905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.238788    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-home\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-home\") pod \"6448b3de-e062-4c1b-be39-1b0871eb4585\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") "
Nov 01 09:58:23.238905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.238860    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6448b3de-e062-4c1b-be39-1b0871eb4585-tls-certs\" (UniqueName: \"kubernetes.io/secret/6448b3de-e062-4c1b-be39-1b0871eb4585-tls-certs\") pod \"6448b3de-e062-4c1b-be39-1b0871eb4585\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") "
Nov 01 09:58:23.239010 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.238902    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-model-cache" pod "6448b3de-e062-4c1b-be39-1b0871eb4585" (UID: "6448b3de-e062-4c1b-be39-1b0871eb4585"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.239010 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.238916    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-tmp-dir\") pod \"6448b3de-e062-4c1b-be39-1b0871eb4585\" (UID: \"6448b3de-e062-4c1b-be39-1b0871eb4585\") "
Nov 01 09:58:23.239136 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.239083    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-home" pod "6448b3de-e062-4c1b-be39-1b0871eb4585" (UID: "6448b3de-e062-4c1b-be39-1b0871eb4585"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.239314 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.239296    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.239537 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.239320    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.239537 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.239324    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-tmp-dir" pod "6448b3de-e062-4c1b-be39-1b0871eb4585" (UID: "6448b3de-e062-4c1b-be39-1b0871eb4585"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.241053 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.241014    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6448b3de-e062-4c1b-be39-1b0871eb4585-tls-certs" pod "6448b3de-e062-4c1b-be39-1b0871eb4585" (UID: "6448b3de-e062-4c1b-be39-1b0871eb4585"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:23.241053 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.241040    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-dshm" pod "6448b3de-e062-4c1b-be39-1b0871eb4585" (UID: "6448b3de-e062-4c1b-be39-1b0871eb4585"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.241163 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.241100    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6448b3de-e062-4c1b-be39-1b0871eb4585-kube-api-access-ch96z" pod "6448b3de-e062-4c1b-be39-1b0871eb4585" (UID: "6448b3de-e062-4c1b-be39-1b0871eb4585"). InnerVolumeSpecName "kube-api-access-ch96z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:23.344053 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.340364    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch96z\" (UniqueName: \"kubernetes.io/projected/6448b3de-e062-4c1b-be39-1b0871eb4585-kube-api-access-ch96z\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.344053 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.340393    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6448b3de-e062-4c1b-be39-1b0871eb4585-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.344053 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.340424    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.344053 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.340433    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6448b3de-e062-4c1b-be39-1b0871eb4585-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.484790 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.484758    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 09:58:23.496863 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.496841    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 09:58:23.521317 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.521295    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 09:58:23.542296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542274    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-home\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-home\") pod \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") "
Nov 01 09:58:23.542428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542309    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-model-cache\") pod \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") "
Nov 01 09:58:23.542428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542332    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tmp-dir\") pod \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") "
Nov 01 09:58:23.542428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542362    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tls-certs\") pod \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") "
Nov 01 09:58:23.542428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542392    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-model-cache\") pod \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") "
Nov 01 09:58:23.542428 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542418    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-kube-api-access-ptj9n\" (UniqueName: \"kubernetes.io/projected/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-kube-api-access-ptj9n\") pod \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542440    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-dshm\") pod \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542495    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-dshm\") pod \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542537    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tmp-dir\") pod \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542570    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-kube-api-access-55nw8\" (UniqueName: \"kubernetes.io/projected/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-kube-api-access-55nw8\") pod \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542571    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-home" pod "26fa7b17-f3ec-4165-b22a-09c4d95837d6" (UID: "26fa7b17-f3ec-4165-b22a-09c4d95837d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542600    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tls-certs\" (UniqueName: \"kubernetes.io/secret/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tls-certs\") pod \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542628    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-home\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-home\") pod \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542660    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-home\") pod \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542596    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-model-cache" pod "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" (UID: "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542696    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fa7b17-f3ec-4165-b22a-09c4d95837d6-kube-api-access-94sx7\" (UniqueName: \"kubernetes.io/projected/26fa7b17-f3ec-4165-b22a-09c4d95837d6-kube-api-access-94sx7\") pod \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542784    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tmp-dir" pod "26fa7b17-f3ec-4165-b22a-09c4d95837d6" (UID: "26fa7b17-f3ec-4165-b22a-09c4d95837d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542800    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-dshm\") pod \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") "
Nov 01 09:58:23.542815 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542810    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tmp-dir" pod "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" (UID: "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.543473 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542875    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-model-cache\") pod \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\" (UID: \"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc\") "
Nov 01 09:58:23.543473 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.542937    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tmp-dir\") pod \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\" (UID: \"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49\") "
Nov 01 09:58:23.543473 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.543038    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tls-certs\") pod \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\" (UID: \"26fa7b17-f3ec-4165-b22a-09c4d95837d6\") "
Nov 01 09:58:23.543473 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.543265    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-home" pod "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" (UID: "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.543473 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.543470    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.543720 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.543489    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.543720 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.543506    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.543720 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.543521    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.543720 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.543536    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.544007 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.543984    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-home" pod "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" (UID: "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.544365 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.544338    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-model-cache" pod "26fa7b17-f3ec-4165-b22a-09c4d95837d6" (UID: "26fa7b17-f3ec-4165-b22a-09c4d95837d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.545477 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.545450    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-model-cache" pod "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" (UID: "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.545652 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.545614    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tls-certs" pod "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" (UID: "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:23.545824 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.545799    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tmp-dir" pod "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" (UID: "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.545924 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.545824    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tls-certs" pod "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" (UID: "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:23.546004 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.545969    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tls-certs" pod "26fa7b17-f3ec-4165-b22a-09c4d95837d6" (UID: "26fa7b17-f3ec-4165-b22a-09c4d95837d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:23.546442 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.546391    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-dshm" pod "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" (UID: "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.546932 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.546909    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-dshm" pod "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" (UID: "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.547453 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.547434    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-kube-api-access-ptj9n" pod "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" (UID: "bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49"). InnerVolumeSpecName "kube-api-access-ptj9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:23.547814 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.547796    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-kube-api-access-55nw8" pod "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" (UID: "41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc"). InnerVolumeSpecName "kube-api-access-55nw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:23.548129 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.548111    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fa7b17-f3ec-4165-b22a-09c4d95837d6-kube-api-access-94sx7" pod "26fa7b17-f3ec-4165-b22a-09c4d95837d6" (UID: "26fa7b17-f3ec-4165-b22a-09c4d95837d6"). InnerVolumeSpecName "kube-api-access-94sx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:23.548371 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.548346    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-dshm" pod "26fa7b17-f3ec-4165-b22a-09c4d95837d6" (UID: "26fa7b17-f3ec-4165-b22a-09c4d95837d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:23.645045 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.644993    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645045 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645045    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645063    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94sx7\" (UniqueName: \"kubernetes.io/projected/26fa7b17-f3ec-4165-b22a-09c4d95837d6-kube-api-access-94sx7\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645078    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645091    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645106    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645121    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa7b17-f3ec-4165-b22a-09c4d95837d6-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645132    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645145    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645159    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptj9n\" (UniqueName: \"kubernetes.io/projected/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-kube-api-access-ptj9n\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645174    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa7b17-f3ec-4165-b22a-09c4d95837d6-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645186    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.645291 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.645198    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55nw8\" (UniqueName: \"kubernetes.io/projected/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc-kube-api-access-55nw8\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:23.755184 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.755098    4931 generic.go:356] "Generic (PLEG): container finished" podID="bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" containerID="ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab" exitCode=0
Nov 01 09:58:23.755184 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.755137    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"
Nov 01 09:58:23.755403 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.755191    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k" event={"ID":"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49","Type":"ContainerDied","Data":"ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab"}
Nov 01 09:58:23.755403 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.755230    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k" event={"ID":"bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49","Type":"ContainerDied","Data":"b73943f138dbd259ce8fd08019bc376e03837d81561993113ad66456e4754b1a"}
Nov 01 09:58:23.755403 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.755272    4931 scope.go:122] "RemoveContainer" containerID="ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab"
Nov 01 09:58:23.757439 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.757411    4931 generic.go:356] "Generic (PLEG): container finished" podID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerID="23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef" exitCode=0
Nov 01 09:58:23.757560 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.757487    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" event={"ID":"26fa7b17-f3ec-4165-b22a-09c4d95837d6","Type":"ContainerDied","Data":"23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef"}
Nov 01 09:58:23.757560 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.757526    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk" event={"ID":"26fa7b17-f3ec-4165-b22a-09c4d95837d6","Type":"ContainerDied","Data":"1c79b21e30ea115646947d45a0d2eac36ab20b441101d0651a11535ffee40a37"}
Nov 01 09:58:23.757659 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.757561    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"
Nov 01 09:58:23.765248 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.765197    4931 generic.go:356] "Generic (PLEG): container finished" podID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerID="d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb" exitCode=0
Nov 01 09:58:23.765366 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.765341    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"
Nov 01 09:58:23.765541 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.765480    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" event={"ID":"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc","Type":"ContainerDied","Data":"d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb"}
Nov 01 09:58:23.765541 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.765517    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" event={"ID":"41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc","Type":"ContainerDied","Data":"d7d183ad88f93435633b6db0088756cfc48f0fec5dc9c99d39d74117e379d958"}
Nov 01 09:58:23.767282 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.767266    4931 scope.go:122] "RemoveContainer" containerID="ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab"
Nov 01 09:58:23.767560 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:58:23.767523    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab\": container with ID starting with ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab not found: ID does not exist" containerID="ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab"
Nov 01 09:58:23.767662 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.767569    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab"} err="failed to get container status \"ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab\": rpc error: code = NotFound desc = could not find container \"ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab\": container with ID starting with ac3c1c259226dbfc805868705a3941d1e27fef2d66e38ad8a606418442f3e6ab not found: ID does not exist"
Nov 01 09:58:23.767662 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.767590    4931 scope.go:122] "RemoveContainer" containerID="23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef"
Nov 01 09:58:23.767781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.767655    4931 generic.go:356] "Generic (PLEG): container finished" podID="6448b3de-e062-4c1b-be39-1b0871eb4585" containerID="2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906" exitCode=0
Nov 01 09:58:23.767781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.767696    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"
Nov 01 09:58:23.767781 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.767769    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6" event={"ID":"6448b3de-e062-4c1b-be39-1b0871eb4585","Type":"ContainerDied","Data":"2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906"}
Nov 01 09:58:23.767962 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.767796    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6" event={"ID":"6448b3de-e062-4c1b-be39-1b0871eb4585","Type":"ContainerDied","Data":"9c479fb27ab68d0e2ae2425aec2bf33345b440344cfeaece4adfe7f5a1e4b4ac"}
Nov 01 09:58:23.780173 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.780156    4931 scope.go:122] "RemoveContainer" containerID="23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef"
Nov 01 09:58:23.780450 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:58:23.780430    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef\": container with ID starting with 23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef not found: ID does not exist" containerID="23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef"
Nov 01 09:58:23.780503 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.780458    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef"} err="failed to get container status \"23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef\": rpc error: code = NotFound desc = could not find container \"23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef\": container with ID starting with 23917bf70c8ce30638b26d582950047a6068c463fb3c819bd680564f50526eef not found: ID does not exist"
Nov 01 09:58:23.780503 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.780473    4931 scope.go:122] "RemoveContainer" containerID="d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb"
Nov 01 09:58:23.790358 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.790341    4931 scope.go:122] "RemoveContainer" containerID="d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb"
Nov 01 09:58:23.790626 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:58:23.790605    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb\": container with ID starting with d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb not found: ID does not exist" containerID="d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb"
Nov 01 09:58:23.790709 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.790637    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb"} err="failed to get container status \"d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb\": rpc error: code = NotFound desc = could not find container \"d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb\": container with ID starting with d7dfb5dfbb483cfa5f0a17b421e2bde10af225db50fab90032e780c3dbd347fb not found: ID does not exist"
Nov 01 09:58:23.790709 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.790660    4931 scope.go:122] "RemoveContainer" containerID="2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906"
Nov 01 09:58:23.799273 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.799187    4931 scope.go:122] "RemoveContainer" containerID="2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906"
Nov 01 09:58:23.799497 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:58:23.799467    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906\": container with ID starting with 2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906 not found: ID does not exist" containerID="2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906"
Nov 01 09:58:23.799630 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.799506    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906"} err="failed to get container status \"2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906\": rpc error: code = NotFound desc = could not find container \"2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906\": container with ID starting with 2492d8fae9a83b4091a54c47b76148dfb9f851eba0d0855595e72ca62052d906 not found: ID does not exist"
Nov 01 09:58:23.937931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.937898    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"]
Nov 01 09:58:23.941067 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:23.941045    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5"]
Nov 01 09:58:24.048986 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:24.048894    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" path="/var/lib/kubelet/pods/41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc/volumes"
Nov 01 09:58:24.049300 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:24.049280    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"]
Nov 01 09:58:24.051842 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:24.051823    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-2-w0-751a73-kserve-8677d98654-kcf8k"]
Nov 01 09:58:24.155198 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:24.155170    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"]
Nov 01 09:58:24.157357 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:24.157335    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-751a73/distinct-w0-751a73-kserve-79f86d5649-2gdqk"]
Nov 01 09:58:24.262112 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:24.262078    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"]
Nov 01 09:58:24.264545 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:24.264524    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/rate-test-model-fd5a10-kserve-69d65bbcfc-p2ht6"]
Nov 01 09:58:24.411884 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:24.411850    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-751a73/simulator-w0-751a73-kserve-7bf9b5679f-xcwb5" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 09:58:26.047632 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:26.047598    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" path="/var/lib/kubelet/pods/26fa7b17-f3ec-4165-b22a-09c4d95837d6/volumes"
Nov 01 09:58:26.048046 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:26.047998    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6448b3de-e062-4c1b-be39-1b0871eb4585" path="/var/lib/kubelet/pods/6448b3de-e062-4c1b-be39-1b0871eb4585/volumes"
Nov 01 09:58:26.048366 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:26.048353    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" path="/var/lib/kubelet/pods/bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49/volumes"
Nov 01 09:58:31.021280 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:58:31.021244    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:58:31.484161 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.484122    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t" podUID="9c142cca-f4af-4f69-823d-43286232c40b" containerName="main" containerID="cri-o://63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085" gracePeriod=45
Nov 01 09:58:31.746839 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.746771    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:58:31.792665 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.792634    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"]
Nov 01 09:58:31.792953 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.792894    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt" podUID="4282f02f-5630-4392-94c3-7f83cdcecc1a" containerName="maas-api" containerID="cri-o://4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826" gracePeriod=30
Nov 01 09:58:31.816680 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.816653    4931 generic.go:356] "Generic (PLEG): container finished" podID="9c142cca-f4af-4f69-823d-43286232c40b" containerID="63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085" exitCode=0
Nov 01 09:58:31.816800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.816705    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"
Nov 01 09:58:31.816800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.816728    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t" event={"ID":"9c142cca-f4af-4f69-823d-43286232c40b","Type":"ContainerDied","Data":"63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085"}
Nov 01 09:58:31.816800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.816768    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t" event={"ID":"9c142cca-f4af-4f69-823d-43286232c40b","Type":"ContainerDied","Data":"5746dc49110488b3c55d59c5b6d40c9925d01750e06ab177a71e44f7223ac6ba"}
Nov 01 09:58:31.816800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.816786    4931 scope.go:122] "RemoveContainer" containerID="63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085"
Nov 01 09:58:31.826855 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.826838    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-home\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-home\") pod \"9c142cca-f4af-4f69-823d-43286232c40b\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") "
Nov 01 09:58:31.826931 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.826902    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c142cca-f4af-4f69-823d-43286232c40b-kube-api-access-vkn4d\" (UniqueName: \"kubernetes.io/projected/9c142cca-f4af-4f69-823d-43286232c40b-kube-api-access-vkn4d\") pod \"9c142cca-f4af-4f69-823d-43286232c40b\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") "
Nov 01 09:58:31.827006 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.826994    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c142cca-f4af-4f69-823d-43286232c40b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c142cca-f4af-4f69-823d-43286232c40b-tls-certs\") pod \"9c142cca-f4af-4f69-823d-43286232c40b\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") "
Nov 01 09:58:31.827102 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.827047    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-tmp-dir\") pod \"9c142cca-f4af-4f69-823d-43286232c40b\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") "
Nov 01 09:58:31.827102 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.827082    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-home" pod "9c142cca-f4af-4f69-823d-43286232c40b" (UID: "9c142cca-f4af-4f69-823d-43286232c40b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:31.827102 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.827093    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-dshm\") pod \"9c142cca-f4af-4f69-823d-43286232c40b\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") "
Nov 01 09:58:31.827252 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.827140    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-model-cache\") pod \"9c142cca-f4af-4f69-823d-43286232c40b\" (UID: \"9c142cca-f4af-4f69-823d-43286232c40b\") "
Nov 01 09:58:31.827732 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.827402    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-tmp-dir" pod "9c142cca-f4af-4f69-823d-43286232c40b" (UID: "9c142cca-f4af-4f69-823d-43286232c40b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:31.827732 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.827464    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-model-cache" pod "9c142cca-f4af-4f69-823d-43286232c40b" (UID: "9c142cca-f4af-4f69-823d-43286232c40b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:31.827732 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.827516    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:31.827732 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.827533    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:31.829606 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.829588    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c142cca-f4af-4f69-823d-43286232c40b-kube-api-access-vkn4d" pod "9c142cca-f4af-4f69-823d-43286232c40b" (UID: "9c142cca-f4af-4f69-823d-43286232c40b"). InnerVolumeSpecName "kube-api-access-vkn4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:31.829796 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.829776    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c142cca-f4af-4f69-823d-43286232c40b-tls-certs" pod "9c142cca-f4af-4f69-823d-43286232c40b" (UID: "9c142cca-f4af-4f69-823d-43286232c40b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:31.829861 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.829781    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-dshm" pod "9c142cca-f4af-4f69-823d-43286232c40b" (UID: "9c142cca-f4af-4f69-823d-43286232c40b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:31.906676 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.906647    4931 scope.go:122] "RemoveContainer" containerID="63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085"
Nov 01 09:58:31.907139 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:58:31.907110    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085\": container with ID starting with 63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085 not found: ID does not exist" containerID="63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085"
Nov 01 09:58:31.907224 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.907154    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085"} err="failed to get container status \"63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085\": rpc error: code = NotFound desc = could not find container \"63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085\": container with ID starting with 63b681ee3ea64ea62de21a752ecbbc6ed719f6c08d344cec612d06bb5dc1b085 not found: ID does not exist"
Nov 01 09:58:31.928385 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.928355    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c142cca-f4af-4f69-823d-43286232c40b-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:31.928385 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.928383    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:31.928385 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.928392    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c142cca-f4af-4f69-823d-43286232c40b-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:31.928682 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:31.928401    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkn4d\" (UniqueName: \"kubernetes.io/projected/9c142cca-f4af-4f69-823d-43286232c40b-kube-api-access-vkn4d\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:32.099280 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.099251    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 09:58:32.130796 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.130767    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4282f02f-5630-4392-94c3-7f83cdcecc1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4282f02f-5630-4392-94c3-7f83cdcecc1a-openshift-service-ca\") pod \"4282f02f-5630-4392-94c3-7f83cdcecc1a\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") "
Nov 01 09:58:32.130939 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.130812    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-metrics-certs\") pod \"4282f02f-5630-4392-94c3-7f83cdcecc1a\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") "
Nov 01 09:58:32.131015 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.130978    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4282f02f-5630-4392-94c3-7f83cdcecc1a-kube-api-access-nzw4l\" (UniqueName: \"kubernetes.io/projected/4282f02f-5630-4392-94c3-7f83cdcecc1a-kube-api-access-nzw4l\") pod \"4282f02f-5630-4392-94c3-7f83cdcecc1a\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") "
Nov 01 09:58:32.131099 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.131074    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls\") pod \"4282f02f-5630-4392-94c3-7f83cdcecc1a\" (UID: \"4282f02f-5630-4392-94c3-7f83cdcecc1a\") "
Nov 01 09:58:32.131151 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.131095    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4282f02f-5630-4392-94c3-7f83cdcecc1a-openshift-service-ca" pod "4282f02f-5630-4392-94c3-7f83cdcecc1a" (UID: "4282f02f-5630-4392-94c3-7f83cdcecc1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:58:32.131490 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.131469    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4282f02f-5630-4392-94c3-7f83cdcecc1a-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:32.133296 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.133265    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4282f02f-5630-4392-94c3-7f83cdcecc1a-kube-api-access-nzw4l" pod "4282f02f-5630-4392-94c3-7f83cdcecc1a" (UID: "4282f02f-5630-4392-94c3-7f83cdcecc1a"). InnerVolumeSpecName "kube-api-access-nzw4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:32.133576 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.133555    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls" pod "4282f02f-5630-4392-94c3-7f83cdcecc1a" (UID: "4282f02f-5630-4392-94c3-7f83cdcecc1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:32.133631 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.133577    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-metrics-certs" pod "4282f02f-5630-4392-94c3-7f83cdcecc1a" (UID: "4282f02f-5630-4392-94c3-7f83cdcecc1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:58:32.232881 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.232847    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzw4l\" (UniqueName: \"kubernetes.io/projected/4282f02f-5630-4392-94c3-7f83cdcecc1a-kube-api-access-nzw4l\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:32.232881 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.232877    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:32.232881 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.232887    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4282f02f-5630-4392-94c3-7f83cdcecc1a-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:32.321013 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.320977    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"]
Nov 01 09:58:32.325865 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.325835    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf99/rate-test-model-46bf99-kserve-b676747f7-zdh4t"]
Nov 01 09:58:32.826946 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.826862    4931 generic.go:356] "Generic (PLEG): container finished" podID="4282f02f-5630-4392-94c3-7f83cdcecc1a" containerID="4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826" exitCode=0
Nov 01 09:58:32.827137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.826957    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"
Nov 01 09:58:32.827137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.826987    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt" event={"ID":"4282f02f-5630-4392-94c3-7f83cdcecc1a","Type":"ContainerDied","Data":"4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826"}
Nov 01 09:58:32.827137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.827069    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt" event={"ID":"4282f02f-5630-4392-94c3-7f83cdcecc1a","Type":"ContainerDied","Data":"a8e88ee6f71a6257c34b68f60b1e8f4bd50c531efcc37bd908d158637648dab6"}
Nov 01 09:58:32.827137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.827109    4931 scope.go:122] "RemoveContainer" containerID="4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826"
Nov 01 09:58:32.837752 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.837716    4931 scope.go:122] "RemoveContainer" containerID="4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826"
Nov 01 09:58:32.837995 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:58:32.837974    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826\": container with ID starting with 4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826 not found: ID does not exist" containerID="4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826"
Nov 01 09:58:32.838137 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.837998    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826"} err="failed to get container status \"4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826\": rpc error: code = NotFound desc = could not find container \"4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826\": container with ID starting with 4c7d16e1906b07f91f55dd5d149d6cdd68d05fa65201786899447bb11b7dc826 not found: ID does not exist"
Nov 01 09:58:32.984806 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.984770    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"]
Nov 01 09:58:32.988970 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:32.988946    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-751a73-5dfccc49fc-lv9wt"]
Nov 01 09:58:34.046856 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:34.046822    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4282f02f-5630-4392-94c3-7f83cdcecc1a" path="/var/lib/kubelet/pods/4282f02f-5630-4392-94c3-7f83cdcecc1a/volumes"
Nov 01 09:58:34.047293 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:34.047278    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c142cca-f4af-4f69-823d-43286232c40b" path="/var/lib/kubelet/pods/9c142cca-f4af-4f69-823d-43286232c40b/volumes"
Nov 01 09:58:49.843662 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:49.843631    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"]
Nov 01 09:58:49.844252 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:49.843829    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" containerID="cri-o://21ac25c6995972713c9730180a61d605c696b675e2798327f4464743edef9a69" gracePeriod=30
Nov 01 09:58:50.504866 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:50.504832    4931 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 09:58:50.505068 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:50.504879    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 09:58:54.975785 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:54.975755    4931 generic.go:356] "Generic (PLEG): container finished" podID="b859b481-69c9-445b-b38f-aefd47600932" containerID="21ac25c6995972713c9730180a61d605c696b675e2798327f4464743edef9a69" exitCode=0
Nov 01 09:58:54.976231 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:54.975836    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" event={"ID":"b859b481-69c9-445b-b38f-aefd47600932","Type":"ContainerDied","Data":"21ac25c6995972713c9730180a61d605c696b675e2798327f4464743edef9a69"}
Nov 01 09:58:55.105938 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.105908    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 09:58:55.272654 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.272570    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b859b481-69c9-445b-b38f-aefd47600932-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b859b481-69c9-445b-b38f-aefd47600932-istio-podinfo\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.272654 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.272636    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-certs\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.272873 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.272739    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-envoy\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.272873 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.272818    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-socket\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.272996 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.272938    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-kube-api-access-jwv7d\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-kube-api-access-jwv7d\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.272996 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.272969    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-istio-token\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-istio-token\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.273123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.272937    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-certs" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:55.273123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273057    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-credential-socket\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.273123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273063    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-socket" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:55.273123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273085    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-data\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.273123 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273120    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b859b481-69c9-445b-b38f-aefd47600932-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b859b481-69c9-445b-b38f-aefd47600932-istiod-ca-cert\") pod \"b859b481-69c9-445b-b38f-aefd47600932\" (UID: \"b859b481-69c9-445b-b38f-aefd47600932\") "
Nov 01 09:58:55.273365 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273265    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-credential-socket" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:55.273580 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273558    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b859b481-69c9-445b-b38f-aefd47600932-istiod-ca-cert" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 09:58:55.273682 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273599    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.273682 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273619    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.273682 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273631    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.273841 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.273794    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-data" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:55.275467 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.275432    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b859b481-69c9-445b-b38f-aefd47600932-istio-podinfo" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 09:58:55.275675 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.275652    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-kube-api-access-jwv7d" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "kube-api-access-jwv7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:55.276122 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.276088    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-istio-token" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:58:55.276244 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.276117    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-envoy" pod "b859b481-69c9-445b-b38f-aefd47600932" (UID: "b859b481-69c9-445b-b38f-aefd47600932"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:58:55.375076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.374983    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwv7d\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-kube-api-access-jwv7d\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.375076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.375018    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b859b481-69c9-445b-b38f-aefd47600932-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.375076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.375051    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.375076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.375064    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b859b481-69c9-445b-b38f-aefd47600932-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.375076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.375078    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b859b481-69c9-445b-b38f-aefd47600932-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.375076 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.375091    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b859b481-69c9-445b-b38f-aefd47600932-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:58:55.983693 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.983661    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"
Nov 01 09:58:55.983693 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.983681    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb" event={"ID":"b859b481-69c9-445b-b38f-aefd47600932","Type":"ContainerDied","Data":"6b64454384fdcd8904534b60f9954b620985f6f7382e979c2e997ff32866612c"}
Nov 01 09:58:55.984179 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:55.983738    4931 scope.go:122] "RemoveContainer" containerID="21ac25c6995972713c9730180a61d605c696b675e2798327f4464743edef9a69"
Nov 01 09:58:56.190886 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:56.190853    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"]
Nov 01 09:58:56.195095 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:56.195070    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-h9jfb"]
Nov 01 09:58:58.048461 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:58:58.048429    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b859b481-69c9-445b-b38f-aefd47600932" path="/var/lib/kubelet/pods/b859b481-69c9-445b-b38f-aefd47600932/volumes"
Nov 01 09:59:01.024248 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:59:01.024211    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:59:31.021771 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:59:31.021738    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 09:59:37.783257 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:37.783220    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"]
Nov 01 09:59:37.831036 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:37.830988    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"]
Nov 01 09:59:48.549395 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.549353    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"]
Nov 01 09:59:48.550042 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.549955    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy"
Nov 01 09:59:48.550042 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.549975    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy"
Nov 01 09:59:48.550042 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.549990    4931 cpu_manager.go:414] "containerMap: removing container" podUID="6448b3de-e062-4c1b-be39-1b0871eb4585" containerName="main"
Nov 01 09:59:48.550042 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.549997    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6448b3de-e062-4c1b-be39-1b0871eb4585" containerName="main"
Nov 01 09:59:48.550042 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550007    4931 cpu_manager.go:414] "containerMap: removing container" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy"
Nov 01 09:59:48.550042 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550013    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550077    4931 cpu_manager.go:414] "containerMap: removing container" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550085    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550097    4931 cpu_manager.go:414] "containerMap: removing container" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550103    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550117    4931 cpu_manager.go:414] "containerMap: removing container" podUID="9c142cca-f4af-4f69-823d-43286232c40b" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550124    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c142cca-f4af-4f69-823d-43286232c40b" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550136    4931 cpu_manager.go:414] "containerMap: removing container" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550143    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550160    4931 cpu_manager.go:414] "containerMap: removing container" podUID="bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550167    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550184    4931 cpu_manager.go:414] "containerMap: removing container" podUID="4282f02f-5630-4392-94c3-7f83cdcecc1a" containerName="maas-api"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550191    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4282f02f-5630-4392-94c3-7f83cdcecc1a" containerName="maas-api"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550207    4931 cpu_manager.go:414] "containerMap: removing container" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550215    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550231    4931 cpu_manager.go:414] "containerMap: removing container" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550240    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550252    4931 cpu_manager.go:414] "containerMap: removing container" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550260    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550272    4931 cpu_manager.go:414] "containerMap: removing container" podUID="e6ebc602-27d4-4f59-b8d0-f5a99d58b973" containerName="maas-api"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550278    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ebc602-27d4-4f59-b8d0-f5a99d58b973" containerName="maas-api"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550287    4931 cpu_manager.go:414] "containerMap: removing container" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550293    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550303    4931 cpu_manager.go:414] "containerMap: removing container" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main"
Nov 01 09:59:48.550381 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550310    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550423    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ebc602-27d4-4f59-b8d0-f5a99d58b973" containerName="maas-api"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550437    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac89622-b6dc-47b0-9b5b-e723e7b4fb55" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550445    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fa7b17-f3ec-4165-b22a-09c4d95837d6" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550454    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a021d984-a9f1-4c70-b6b5-c4d924d27e99" containerName="istio-proxy"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550463    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b859b481-69c9-445b-b38f-aefd47600932" containerName="istio-proxy"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550474    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6448b3de-e062-4c1b-be39-1b0871eb4585" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550486    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd69395-69b6-443c-8023-9a83841d9aa6" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550494    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a1f4a0-3751-45de-9fc7-89188a792f50" containerName="istio-proxy"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550504    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c142cca-f4af-4f69-823d-43286232c40b" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550514    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8971d8cb-bc36-424c-8523-3b31df38f235" containerName="istio-proxy"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550524    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1ce16d-b00e-4a68-8b2b-7e3ec9a6fa49" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550534    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4282f02f-5630-4392-94c3-7f83cdcecc1a" containerName="maas-api"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550543    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d7b4cb-e2d0-4f0c-a1e6-0b4a887acdcc" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550551    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5496b80d-d6ec-4409-8b8e-eea80d5b1d01" containerName="main"
Nov 01 09:59:48.551520 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.550559    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d12efcf-78f3-424b-ab14-e0f8f756e930" containerName="main"
Nov 01 09:59:48.553627 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.553607    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.603711 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.603685    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.603848 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.603716    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.603848 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.603751    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.603956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.603845    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcv7\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-kube-api-access-dwcv7\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.603956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.603870    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.603956 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.603940    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.604125 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.603985    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684f345-16f3-4f74-9453-047f70f68101-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.604125 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.604047    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684f345-16f3-4f74-9453-047f70f68101-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.604125 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.604057    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a87a1d-openshift-default-dockercfg-jwv8b\""
Nov 01 09:59:48.604258 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.604124    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.655449 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.655419    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"]
Nov 01 09:59:48.704806 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.704774    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.704806 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.704804    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.704851    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.704881    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcv7\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-kube-api-access-dwcv7\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.704917    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705105 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.704945    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705332 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.705133    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684f345-16f3-4f74-9453-047f70f68101-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705332 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.705178    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684f345-16f3-4f74-9453-047f70f68101-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705332 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.705209    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705332 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.705217    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705332 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.705224    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705581 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.705403    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705581 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.705541    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.705962 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.705940    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684f345-16f3-4f74-9453-047f70f68101-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.707282 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.707263    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.707754 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.707734    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684f345-16f3-4f74-9453-047f70f68101-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.763800 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.763775    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.763916 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.763901    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcv7\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-kube-api-access-dwcv7\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:48.867191 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:48.867161    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:49.293600 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:49.293570    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"]
Nov 01 09:59:49.294254 ip-10-0-143-244 kubenswrapper[4931]: W1101 09:59:49.294224    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3684f345_16f3_4f74_9453_047f70f68101.slice/crio-d560a75ac21c5ff47c640bc2ed3775f512060bf2afa40533277530b5b0d4785e WatchSource:0}: Error finding container d560a75ac21c5ff47c640bc2ed3775f512060bf2afa40533277530b5b0d4785e: Status 404 returned error can't find the container with id d560a75ac21c5ff47c640bc2ed3775f512060bf2afa40533277530b5b0d4785e
Nov 01 09:59:49.296464 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:49.296434    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:59:49.296568 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:49.296493    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:59:49.296568 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:49.296523    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 09:59:50.299051 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:50.298988    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" event={"ID":"3684f345-16f3-4f74-9453-047f70f68101","Type":"ContainerStarted","Data":"2ba11a33d17e8d8c1d6e7a9929761a1ccd5a52e33dd47f3052f2cfddbea86964"}
Nov 01 09:59:50.299498 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:50.299071    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" event={"ID":"3684f345-16f3-4f74-9453-047f70f68101","Type":"ContainerStarted","Data":"d560a75ac21c5ff47c640bc2ed3775f512060bf2afa40533277530b5b0d4785e"}
Nov 01 09:59:50.478328 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:50.478257    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" podStartSLOduration=2.478237631 podStartE2EDuration="2.478237631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:59:50.477314973 +0000 UTC m=+5260.592187942" watchObservedRunningTime="2026-11-01 09:59:50.478237631 +0000 UTC m=+5260.593110596"
Nov 01 09:59:50.871047 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:50.869704    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:50.871047 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:50.869831    4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 09:59:50.871047 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:50.869873    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 09:59:51.871449 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:51.871409    4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 09:59:51.872078 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:51.872044    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 09:59:52.803504 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:52.803466    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main" containerID="cri-o://237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551" gracePeriod=45
Nov 01 09:59:52.850511 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:52.850481    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" containerID="cri-o://b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49" gracePeriod=45
Nov 01 09:59:52.874788 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:52.874383    4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 09:59:52.874788 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:52.874441    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 09:59:52.956208 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:52.956174    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused"
Nov 01 09:59:53.156923 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.156893    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 09:59:53.199143 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.199120    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 09:59:53.251109 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251073    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tls-certs\") pod \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") "
Nov 01 09:59:53.251285 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251122    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-tmp-dir\") pod \"b29e67df-614a-4389-8610-8bfadbfa0a91\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") "
Nov 01 09:59:53.251285 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251145    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-home\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-home\") pod \"b29e67df-614a-4389-8610-8bfadbfa0a91\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") "
Nov 01 09:59:53.251285 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251188    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-model-cache\") pod \"b29e67df-614a-4389-8610-8bfadbfa0a91\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") "
Nov 01 09:59:53.251285 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251226    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-dshm\") pod \"b29e67df-614a-4389-8610-8bfadbfa0a91\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") "
Nov 01 09:59:53.251285 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251271    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9d0733-3c83-462e-bbd3-c2be6acd5327-kube-api-access-n2m6m\" (UniqueName: \"kubernetes.io/projected/cb9d0733-3c83-462e-bbd3-c2be6acd5327-kube-api-access-n2m6m\") pod \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") "
Nov 01 09:59:53.251549 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251341    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b29e67df-614a-4389-8610-8bfadbfa0a91-tls-certs\" (UniqueName: \"kubernetes.io/secret/b29e67df-614a-4389-8610-8bfadbfa0a91-tls-certs\") pod \"b29e67df-614a-4389-8610-8bfadbfa0a91\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") "
Nov 01 09:59:53.251549 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251417    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-tmp-dir" pod "b29e67df-614a-4389-8610-8bfadbfa0a91" (UID: "b29e67df-614a-4389-8610-8bfadbfa0a91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.251657 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251616    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-home" pod "b29e67df-614a-4389-8610-8bfadbfa0a91" (UID: "b29e67df-614a-4389-8610-8bfadbfa0a91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.251657 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251631    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-model-cache" pod "b29e67df-614a-4389-8610-8bfadbfa0a91" (UID: "b29e67df-614a-4389-8610-8bfadbfa0a91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.251988 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.251386    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-dshm\") pod \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") "
Nov 01 09:59:53.252219 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252200    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-home\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-home\") pod \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") "
Nov 01 09:59:53.252309 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252268    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29e67df-614a-4389-8610-8bfadbfa0a91-kube-api-access-flm8q\" (UniqueName: \"kubernetes.io/projected/b29e67df-614a-4389-8610-8bfadbfa0a91-kube-api-access-flm8q\") pod \"b29e67df-614a-4389-8610-8bfadbfa0a91\" (UID: \"b29e67df-614a-4389-8610-8bfadbfa0a91\") "
Nov 01 09:59:53.252309 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252305    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tmp-dir\") pod \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") "
Nov 01 09:59:53.252424 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252358    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-model-cache\") pod \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\" (UID: \"cb9d0733-3c83-462e-bbd3-c2be6acd5327\") "
Nov 01 09:59:53.252509 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252484    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-home" pod "cb9d0733-3c83-462e-bbd3-c2be6acd5327" (UID: "cb9d0733-3c83-462e-bbd3-c2be6acd5327"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.252761 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252686    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tmp-dir" pod "cb9d0733-3c83-462e-bbd3-c2be6acd5327" (UID: "cb9d0733-3c83-462e-bbd3-c2be6acd5327"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.252761 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252744    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-model-cache" pod "cb9d0733-3c83-462e-bbd3-c2be6acd5327" (UID: "cb9d0733-3c83-462e-bbd3-c2be6acd5327"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.252761 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252760    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.253009 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252779    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.253009 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252793    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.253009 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.252806    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.254407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.254311    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-dshm" pod "b29e67df-614a-4389-8610-8bfadbfa0a91" (UID: "b29e67df-614a-4389-8610-8bfadbfa0a91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.254407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.254380    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tls-certs" pod "cb9d0733-3c83-462e-bbd3-c2be6acd5327" (UID: "cb9d0733-3c83-462e-bbd3-c2be6acd5327"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:59:53.254407 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.254392    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9d0733-3c83-462e-bbd3-c2be6acd5327-kube-api-access-n2m6m" pod "cb9d0733-3c83-462e-bbd3-c2be6acd5327" (UID: "cb9d0733-3c83-462e-bbd3-c2be6acd5327"). InnerVolumeSpecName "kube-api-access-n2m6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:53.254565 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.254497    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29e67df-614a-4389-8610-8bfadbfa0a91-tls-certs" pod "b29e67df-614a-4389-8610-8bfadbfa0a91" (UID: "b29e67df-614a-4389-8610-8bfadbfa0a91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 09:59:53.254693 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.254675    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29e67df-614a-4389-8610-8bfadbfa0a91-kube-api-access-flm8q" pod "b29e67df-614a-4389-8610-8bfadbfa0a91" (UID: "b29e67df-614a-4389-8610-8bfadbfa0a91"). InnerVolumeSpecName "kube-api-access-flm8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 09:59:53.255190 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.255173    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-dshm" pod "cb9d0733-3c83-462e-bbd3-c2be6acd5327" (UID: "cb9d0733-3c83-462e-bbd3-c2be6acd5327"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 09:59:53.336180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.332341    4931 generic.go:356] "Generic (PLEG): container finished" podID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerID="237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551" exitCode=0
Nov 01 09:59:53.336180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.332470    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"
Nov 01 09:59:53.336180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.332826    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" event={"ID":"b29e67df-614a-4389-8610-8bfadbfa0a91","Type":"ContainerDied","Data":"237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551"}
Nov 01 09:59:53.336180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.332870    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd" event={"ID":"b29e67df-614a-4389-8610-8bfadbfa0a91","Type":"ContainerDied","Data":"558e8693601a3c16ccded6e8bea48a154e2bfdecf5d840c736df20690ffbf1a7"}
Nov 01 09:59:53.336180 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.332891    4931 scope.go:122] "RemoveContainer" containerID="237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551"
Nov 01 09:59:53.339958 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.338848    4931 generic.go:356] "Generic (PLEG): container finished" podID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerID="b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49" exitCode=0
Nov 01 09:59:53.339958 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.338949    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" event={"ID":"cb9d0733-3c83-462e-bbd3-c2be6acd5327","Type":"ContainerDied","Data":"b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49"}
Nov 01 09:59:53.339958 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.338972    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz" event={"ID":"cb9d0733-3c83-462e-bbd3-c2be6acd5327","Type":"ContainerDied","Data":"57c41652dedaabe7f73a0118ecb4f3612e205610bb861d843309681ad0ced549"}
Nov 01 09:59:53.339958 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.339015    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"
Nov 01 09:59:53.349622 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.349602    4931 scope.go:122] "RemoveContainer" containerID="237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551"
Nov 01 09:59:53.349872 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:59:53.349854    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551\": container with ID starting with 237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551 not found: ID does not exist" containerID="237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551"
Nov 01 09:59:53.349919 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.349882    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551"} err="failed to get container status \"237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551\": rpc error: code = NotFound desc = could not find container \"237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551\": container with ID starting with 237d9edb1abf7971c9f81f35a053757d3135a264a691b87e1f375ecabe594551 not found: ID does not exist"
Nov 01 09:59:53.349919 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.349898    4931 scope.go:122] "RemoveContainer" containerID="b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49"
Nov 01 09:59:53.353721 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.353703    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29e67df-614a-4389-8610-8bfadbfa0a91-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.353905 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.353857    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.354040 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.354011    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flm8q\" (UniqueName: \"kubernetes.io/projected/b29e67df-614a-4389-8610-8bfadbfa0a91-kube-api-access-flm8q\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.354136 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.354123    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.354308 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.354298    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9d0733-3c83-462e-bbd3-c2be6acd5327-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.354389 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.354381    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9d0733-3c83-462e-bbd3-c2be6acd5327-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.354439 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.354432    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29e67df-614a-4389-8610-8bfadbfa0a91-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.354486 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.354478    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2m6m\" (UniqueName: \"kubernetes.io/projected/cb9d0733-3c83-462e-bbd3-c2be6acd5327-kube-api-access-n2m6m\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 09:59:53.358247 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.358234    4931 scope.go:122] "RemoveContainer" containerID="b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49"
Nov 01 09:59:53.358587 ip-10-0-143-244 kubenswrapper[4931]: E1101 09:59:53.358567    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49\": container with ID starting with b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49 not found: ID does not exist" containerID="b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49"
Nov 01 09:59:53.358659 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.358594    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49"} err="failed to get container status \"b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49\": rpc error: code = NotFound desc = could not find container \"b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49\": container with ID starting with b5329c4f69d60e778693738e1dce620ae1f0a417aa87d7e744ea24c6fd2a1d49 not found: ID does not exist"
Nov 01 09:59:53.512564 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.512483    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"]
Nov 01 09:59:53.516518 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.516493    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/simulator-w4-b711ae-kserve-549dc8946-wjwhz"]
Nov 01 09:59:53.629753 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.629726    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"]
Nov 01 09:59:53.635684 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.635660    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b711ae/distinct-2-w4-b711ae-kserve-7678bc86b5-m6qvd"]
Nov 01 09:59:53.879043 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.878392    4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 09:59:53.879043 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:53.878447    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 09:59:54.047345 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:54.047310    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" path="/var/lib/kubelet/pods/b29e67df-614a-4389-8610-8bfadbfa0a91/volumes"
Nov 01 09:59:54.047769 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:54.047755    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" path="/var/lib/kubelet/pods/cb9d0733-3c83-462e-bbd3-c2be6acd5327/volumes"
Nov 01 09:59:54.883047 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:54.872355    4931 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 09:59:54.883047 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:54.872426    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 09:59:56.497392 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:56.497355    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:56.498448 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:56.498418    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 09:59:56.499049 ip-10-0-143-244 kubenswrapper[4931]: I1101 09:59:56.499006    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 10:00:00.166972 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.166935    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"]
Nov 01 10:00:00.167409 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.167394    4931 cpu_manager.go:414] "containerMap: removing container" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main"
Nov 01 10:00:00.167452 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.167410    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main"
Nov 01 10:00:00.167452 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.167431    4931 cpu_manager.go:414] "containerMap: removing container" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main"
Nov 01 10:00:00.167452 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.167436    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main"
Nov 01 10:00:00.167548 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.167497    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9d0733-3c83-462e-bbd3-c2be6acd5327" containerName="main"
Nov 01 10:00:00.167548 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.167511    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29e67df-614a-4389-8610-8bfadbfa0a91" containerName="main"
Nov 01 10:00:00.172239 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.172217    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"]
Nov 01 10:00:00.172376 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.172343    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:00.175689 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.175670    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:00.223774 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.223744    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\""
Nov 01 10:00:00.237148 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.237120    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ee7db7-bf1a-44d5-98e5-fa33478442b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz\" (UID: \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:00.237335 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.237161    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsjxb\" (UniqueName: \"kubernetes.io/projected/02ee7db7-bf1a-44d5-98e5-fa33478442b9-kube-api-access-nsjxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz\" (UID: \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:00.273969 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.273915    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"]
Nov 01 10:00:00.276422 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.276395    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"]
Nov 01 10:00:00.338397 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.338362    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ee7db7-bf1a-44d5-98e5-fa33478442b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz\" (UID: \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:00.338609 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.338409    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsjxb\" (UniqueName: \"kubernetes.io/projected/02ee7db7-bf1a-44d5-98e5-fa33478442b9-kube-api-access-nsjxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz\" (UID: \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:00.338609 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.338435    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5\" (UID: \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:00.338609 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.338484    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhv9t\" (UniqueName: \"kubernetes.io/projected/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-kube-api-access-xhv9t\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5\" (UID: \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:00.339097 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.339076    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ee7db7-bf1a-44d5-98e5-fa33478442b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz\" (UID: \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:00.403291 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.403261    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsjxb\" (UniqueName: \"kubernetes.io/projected/02ee7db7-bf1a-44d5-98e5-fa33478442b9-kube-api-access-nsjxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz\" (UID: \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:00.439631 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.439536    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhv9t\" (UniqueName: \"kubernetes.io/projected/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-kube-api-access-xhv9t\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5\" (UID: \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:00.439828 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.439650    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5\" (UID: \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:00.440254 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.440235    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5\" (UID: \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:00.487477 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.487443    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:00.498719 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.498693    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhv9t\" (UniqueName: \"kubernetes.io/projected/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-kube-api-access-xhv9t\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5\" (UID: \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:00.793543 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.793473    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:00.863313 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:00.862472    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"]
Nov 01 10:00:00.864354 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:00:00.864312    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ee7db7_bf1a_44d5_98e5_fa33478442b9.slice/crio-ce210d9ffd71188fea4dabffcf844f599823a904397ad1e818a22530babd86fd WatchSource:0}: Error finding container ce210d9ffd71188fea4dabffcf844f599823a904397ad1e818a22530babd86fd: Status 404 returned error can't find the container with id ce210d9ffd71188fea4dabffcf844f599823a904397ad1e818a22530babd86fd
Nov 01 10:00:01.022020 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:01.021983    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:00:01.161530 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:01.161497    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"]
Nov 01 10:00:01.162779 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:00:01.162751    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf3e783_2393_4ab0_9dd9_3d0fb3298e15.slice/crio-a7e0091dfb9436c5c0cc541d7a18f69fb9094146731e6cf5c9ee85288cd21e3a WatchSource:0}: Error finding container a7e0091dfb9436c5c0cc541d7a18f69fb9094146731e6cf5c9ee85288cd21e3a: Status 404 returned error can't find the container with id a7e0091dfb9436c5c0cc541d7a18f69fb9094146731e6cf5c9ee85288cd21e3a
Nov 01 10:00:01.388536 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:01.388507    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" event={"ID":"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15","Type":"ContainerStarted","Data":"a7e0091dfb9436c5c0cc541d7a18f69fb9094146731e6cf5c9ee85288cd21e3a"}
Nov 01 10:00:01.390251 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:01.390221    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz" event={"ID":"02ee7db7-bf1a-44d5-98e5-fa33478442b9","Type":"ContainerStarted","Data":"aaf8075c74ce29c9915fc4694d9391a9b259b6d81894ffea3d48e2ed56f64050"}
Nov 01 10:00:01.390410 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:01.390267    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz" event={"ID":"02ee7db7-bf1a-44d5-98e5-fa33478442b9","Type":"ContainerStarted","Data":"ce210d9ffd71188fea4dabffcf844f599823a904397ad1e818a22530babd86fd"}
Nov 01 10:00:01.501806 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:01.501757    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz" podStartSLOduration=1.501739398 podStartE2EDuration="1.501739398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:00:01.501226106 +0000 UTC m=+5271.616099073" watchObservedRunningTime="2026-11-01 10:00:01.501739398 +0000 UTC m=+5271.616612366"
Nov 01 10:00:02.397620 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:02.397588    4931 generic.go:356] "Generic (PLEG): container finished" podID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerID="11ffd1990f092f5d0af4539cca56f2219c7582ffb3c8f6e90ffaa26ea9008d3f" exitCode=60
Nov 01 10:00:02.398103 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:02.397667    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" event={"ID":"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15","Type":"ContainerDied","Data":"11ffd1990f092f5d0af4539cca56f2219c7582ffb3c8f6e90ffaa26ea9008d3f"}
Nov 01 10:00:02.398103 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:02.397841    4931 scope.go:122] "RemoveContainer" containerID="11ffd1990f092f5d0af4539cca56f2219c7582ffb3c8f6e90ffaa26ea9008d3f"
Nov 01 10:00:02.399334 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:02.399311    4931 generic.go:356] "Generic (PLEG): container finished" podID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" containerID="aaf8075c74ce29c9915fc4694d9391a9b259b6d81894ffea3d48e2ed56f64050" exitCode=60
Nov 01 10:00:02.399396 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:02.399380    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz" event={"ID":"02ee7db7-bf1a-44d5-98e5-fa33478442b9","Type":"ContainerDied","Data":"aaf8075c74ce29c9915fc4694d9391a9b259b6d81894ffea3d48e2ed56f64050"}
Nov 01 10:00:02.399530 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:02.399515    4931 scope.go:122] "RemoveContainer" containerID="aaf8075c74ce29c9915fc4694d9391a9b259b6d81894ffea3d48e2ed56f64050"
Nov 01 10:00:03.406411 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:03.406376    4931 generic.go:356] "Generic (PLEG): container finished" podID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerID="1c1be29be0359ddbdc23780e537f004872162a93c919eae63b6543ffa515f6c0" exitCode=60
Nov 01 10:00:03.406876 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:03.406456    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" event={"ID":"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15","Type":"ContainerDied","Data":"1c1be29be0359ddbdc23780e537f004872162a93c919eae63b6543ffa515f6c0"}
Nov 01 10:00:03.406876 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:03.406519    4931 scope.go:122] "RemoveContainer" containerID="11ffd1990f092f5d0af4539cca56f2219c7582ffb3c8f6e90ffaa26ea9008d3f"
Nov 01 10:00:03.406876 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:03.406734    4931 scope.go:122] "RemoveContainer" containerID="1c1be29be0359ddbdc23780e537f004872162a93c919eae63b6543ffa515f6c0"
Nov 01 10:00:03.407068 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:03.407016    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5_odh-ai-gateway-infra(5cf3e783-2393-4ab0-9dd9-3d0fb3298e15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15"
Nov 01 10:00:03.408611 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:03.408587    4931 generic.go:356] "Generic (PLEG): container finished" podID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" containerID="0d7777b7838e7a2d4b2c155b432fa1a4d1679eb17ef15c29f0c8207a81efcf42" exitCode=60
Nov 01 10:00:03.408722 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:03.408664    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz" event={"ID":"02ee7db7-bf1a-44d5-98e5-fa33478442b9","Type":"ContainerDied","Data":"0d7777b7838e7a2d4b2c155b432fa1a4d1679eb17ef15c29f0c8207a81efcf42"}
Nov 01 10:00:03.408867 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:03.408856    4931 scope.go:122] "RemoveContainer" containerID="0d7777b7838e7a2d4b2c155b432fa1a4d1679eb17ef15c29f0c8207a81efcf42"
Nov 01 10:00:03.409086 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:03.409069    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz_odh-ai-gateway-infra(02ee7db7-bf1a-44d5-98e5-fa33478442b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9"
Nov 01 10:00:03.418559 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:03.418544    4931 scope.go:122] "RemoveContainer" containerID="aaf8075c74ce29c9915fc4694d9391a9b259b6d81894ffea3d48e2ed56f64050"
Nov 01 10:00:04.416520 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:04.416491    4931 scope.go:122] "RemoveContainer" containerID="1c1be29be0359ddbdc23780e537f004872162a93c919eae63b6543ffa515f6c0"
Nov 01 10:00:04.416938 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:04.416746    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5_odh-ai-gateway-infra(5cf3e783-2393-4ab0-9dd9-3d0fb3298e15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15"
Nov 01 10:00:04.417991 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:04.417973    4931 scope.go:122] "RemoveContainer" containerID="0d7777b7838e7a2d4b2c155b432fa1a4d1679eb17ef15c29f0c8207a81efcf42"
Nov 01 10:00:04.418204 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:04.418185    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz_odh-ai-gateway-infra(02ee7db7-bf1a-44d5-98e5-fa33478442b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9"
Nov 01 10:00:05.385879 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.385849    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"]
Nov 01 10:00:05.386080 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.386057    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h" podUID="4b00f5ed-abe0-4621-a8a0-530838e68b7c" containerName="maas-api" containerID="cri-o://c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a" gracePeriod=30
Nov 01 10:00:05.474175 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.474128    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"]
Nov 01 10:00:05.645902 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.645847    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:05.716658 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.716633    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 10:00:05.790507 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.790473    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ee7db7-bf1a-44d5-98e5-fa33478442b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ee7db7-bf1a-44d5-98e5-fa33478442b9-maas-api-service-ca\") pod \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\" (UID: \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\") "
Nov 01 10:00:05.790702 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.790542    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ee7db7-bf1a-44d5-98e5-fa33478442b9-kube-api-access-nsjxb\" (UniqueName: \"kubernetes.io/projected/02ee7db7-bf1a-44d5-98e5-fa33478442b9-kube-api-access-nsjxb\") pod \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\" (UID: \"02ee7db7-bf1a-44d5-98e5-fa33478442b9\") "
Nov 01 10:00:05.790702 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.790682    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b00f5ed-abe0-4621-a8a0-530838e68b7c-kube-api-access-2j2rs\" (UniqueName: \"kubernetes.io/projected/4b00f5ed-abe0-4621-a8a0-530838e68b7c-kube-api-access-2j2rs\") pod \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") "
Nov 01 10:00:05.790883 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.790855    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-metrics-certs\") pod \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") "
Nov 01 10:00:05.791009 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.790940    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ee7db7-bf1a-44d5-98e5-fa33478442b9-maas-api-service-ca" pod "02ee7db7-bf1a-44d5-98e5-fa33478442b9" (UID: "02ee7db7-bf1a-44d5-98e5-fa33478442b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:00:05.791269 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.791221    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls\") pod \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") "
Nov 01 10:00:05.791269 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.791289    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b00f5ed-abe0-4621-a8a0-530838e68b7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b00f5ed-abe0-4621-a8a0-530838e68b7c-openshift-service-ca\") pod \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\" (UID: \"4b00f5ed-abe0-4621-a8a0-530838e68b7c\") "
Nov 01 10:00:05.791681 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.791650    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b00f5ed-abe0-4621-a8a0-530838e68b7c-openshift-service-ca" pod "4b00f5ed-abe0-4621-a8a0-530838e68b7c" (UID: "4b00f5ed-abe0-4621-a8a0-530838e68b7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:00:05.791773 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.791713    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ee7db7-bf1a-44d5-98e5-fa33478442b9-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:05.793290 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.793264    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b00f5ed-abe0-4621-a8a0-530838e68b7c-kube-api-access-2j2rs" pod "4b00f5ed-abe0-4621-a8a0-530838e68b7c" (UID: "4b00f5ed-abe0-4621-a8a0-530838e68b7c"). InnerVolumeSpecName "kube-api-access-2j2rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:00:05.793393 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.793350    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ee7db7-bf1a-44d5-98e5-fa33478442b9-kube-api-access-nsjxb" pod "02ee7db7-bf1a-44d5-98e5-fa33478442b9" (UID: "02ee7db7-bf1a-44d5-98e5-fa33478442b9"). InnerVolumeSpecName "kube-api-access-nsjxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:00:05.793680 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.793655    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls" pod "4b00f5ed-abe0-4621-a8a0-530838e68b7c" (UID: "4b00f5ed-abe0-4621-a8a0-530838e68b7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:00:05.793768 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.793750    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-metrics-certs" pod "4b00f5ed-abe0-4621-a8a0-530838e68b7c" (UID: "4b00f5ed-abe0-4621-a8a0-530838e68b7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:00:05.893290 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.893254    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsjxb\" (UniqueName: \"kubernetes.io/projected/02ee7db7-bf1a-44d5-98e5-fa33478442b9-kube-api-access-nsjxb\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:05.893290 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.893284    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j2rs\" (UniqueName: \"kubernetes.io/projected/4b00f5ed-abe0-4621-a8a0-530838e68b7c-kube-api-access-2j2rs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:05.893499 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.893299    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:05.893499 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.893312    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b00f5ed-abe0-4621-a8a0-530838e68b7c-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:05.893499 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:05.893323    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b00f5ed-abe0-4621-a8a0-530838e68b7c-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:06.431086 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.431053    4931 generic.go:356] "Generic (PLEG): container finished" podID="4b00f5ed-abe0-4621-a8a0-530838e68b7c" containerID="c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a" exitCode=0
Nov 01 10:00:06.431244 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.431104    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"
Nov 01 10:00:06.431244 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.431117    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h" event={"ID":"4b00f5ed-abe0-4621-a8a0-530838e68b7c","Type":"ContainerDied","Data":"c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a"}
Nov 01 10:00:06.431244 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.431180    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h" event={"ID":"4b00f5ed-abe0-4621-a8a0-530838e68b7c","Type":"ContainerDied","Data":"fd8b3a801ee95b7df6f491487b6048255e7985abb1466706f4f2997b592d0eb0"}
Nov 01 10:00:06.431244 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.431209    4931 scope.go:122] "RemoveContainer" containerID="c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a"
Nov 01 10:00:06.432757 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.432664    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz" event={"ID":"02ee7db7-bf1a-44d5-98e5-fa33478442b9","Type":"ContainerDied","Data":"ce210d9ffd71188fea4dabffcf844f599823a904397ad1e818a22530babd86fd"}
Nov 01 10:00:06.432757 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.432714    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"
Nov 01 10:00:06.440755 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.440736    4931 scope.go:122] "RemoveContainer" containerID="c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a"
Nov 01 10:00:06.441014 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:06.440995    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a\": container with ID starting with c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a not found: ID does not exist" containerID="c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a"
Nov 01 10:00:06.441105 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.441046    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a"} err="failed to get container status \"c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a\": rpc error: code = NotFound desc = could not find container \"c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a\": container with ID starting with c0ca1316a2e163a0368cd03286bc26c29d26bdd5443b29ea4f7c095e9cab8f6a not found: ID does not exist"
Nov 01 10:00:06.441105 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.441064    4931 scope.go:122] "RemoveContainer" containerID="0d7777b7838e7a2d4b2c155b432fa1a4d1679eb17ef15c29f0c8207a81efcf42"
Nov 01 10:00:06.589936 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.589904    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"]
Nov 01 10:00:06.597273 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.597249    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b711ae-b4dd56db6-6q95h"]
Nov 01 10:00:06.709515 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.709423    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"]
Nov 01 10:00:06.710852 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:06.710826    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892120-ng4fz"]
Nov 01 10:00:08.046864 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:08.046832    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" path="/var/lib/kubelet/pods/02ee7db7-bf1a-44d5-98e5-fa33478442b9/volumes"
Nov 01 10:00:08.047283 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:08.047250    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b00f5ed-abe0-4621-a8a0-530838e68b7c" path="/var/lib/kubelet/pods/4b00f5ed-abe0-4621-a8a0-530838e68b7c/volumes"
Nov 01 10:00:10.502494 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.502465    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"]
Nov 01 10:00:10.502893 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.502886    4931 cpu_manager.go:414] "containerMap: removing container" podUID="4b00f5ed-abe0-4621-a8a0-530838e68b7c" containerName="maas-api"
Nov 01 10:00:10.502949 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.502897    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b00f5ed-abe0-4621-a8a0-530838e68b7c" containerName="maas-api"
Nov 01 10:00:10.502949 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.502907    4931 cpu_manager.go:414] "containerMap: removing container" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" containerName="cleanup"
Nov 01 10:00:10.502949 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.502913    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" containerName="cleanup"
Nov 01 10:00:10.502949 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.502928    4931 cpu_manager.go:414] "containerMap: removing container" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" containerName="cleanup"
Nov 01 10:00:10.502949 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.502932    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" containerName="cleanup"
Nov 01 10:00:10.503133 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.502994    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b00f5ed-abe0-4621-a8a0-530838e68b7c" containerName="maas-api"
Nov 01 10:00:10.503133 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.503003    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" containerName="cleanup"
Nov 01 10:00:10.503133 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.503009    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ee7db7-bf1a-44d5-98e5-fa33478442b9" containerName="cleanup"
Nov 01 10:00:10.507806 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.507782    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.558501 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.558472    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 10:00:10.561685 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.561657    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"]
Nov 01 10:00:10.640245 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.640218    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpvb\" (UniqueName: \"kubernetes.io/projected/7723e8a2-0ece-4771-bfb3-f6b6585ee894-kube-api-access-5cpvb\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.640245 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.640248    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.640439 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.640270    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.741281 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.741253    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpvb\" (UniqueName: \"kubernetes.io/projected/7723e8a2-0ece-4771-bfb3-f6b6585ee894-kube-api-access-5cpvb\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.741442 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.741288    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.741442 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.741308    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.741884 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.741859    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.743836 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.743806    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.801068 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.800973    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpvb\" (UniqueName: \"kubernetes.io/projected/7723e8a2-0ece-4771-bfb3-f6b6585ee894-kube-api-access-5cpvb\") pod \"maas-api-revoke-sub-215b4574-xg2l4\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:10.820365 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:10.820337    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:11.194563 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:11.194527    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"]
Nov 01 10:00:11.195631 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:00:11.195604    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7723e8a2_0ece_4771_bfb3_f6b6585ee894.slice/crio-36ed8d451ff265db2f9b4d1b3763a7594f76e3bcf7dbef54f6cd30d2561d8dda WatchSource:0}: Error finding container 36ed8d451ff265db2f9b4d1b3763a7594f76e3bcf7dbef54f6cd30d2561d8dda: Status 404 returned error can't find the container with id 36ed8d451ff265db2f9b4d1b3763a7594f76e3bcf7dbef54f6cd30d2561d8dda
Nov 01 10:00:11.461416 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:11.461333    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4" event={"ID":"7723e8a2-0ece-4771-bfb3-f6b6585ee894","Type":"ContainerStarted","Data":"36ed8d451ff265db2f9b4d1b3763a7594f76e3bcf7dbef54f6cd30d2561d8dda"}
Nov 01 10:00:12.477460 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:12.477358    4931 generic.go:356] "Generic (PLEG): container finished" podID="7723e8a2-0ece-4771-bfb3-f6b6585ee894" containerID="2702b750d13ccf9a3f619d414db61fdc10b3eb5f72d169a355c776c131e27af2" exitCode=0
Nov 01 10:00:12.477921 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:12.477458    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4" event={"ID":"7723e8a2-0ece-4771-bfb3-f6b6585ee894","Type":"ContainerDied","Data":"2702b750d13ccf9a3f619d414db61fdc10b3eb5f72d169a355c776c131e27af2"}
Nov 01 10:00:13.617732 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.617703    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:13.771325 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.771245    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7723e8a2-0ece-4771-bfb3-f6b6585ee894-kube-api-access-5cpvb\" (UniqueName: \"kubernetes.io/projected/7723e8a2-0ece-4771-bfb3-f6b6585ee894-kube-api-access-5cpvb\") pod \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") "
Nov 01 10:00:13.771325 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.771316    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-xks-ca\") pod \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") "
Nov 01 10:00:13.771540 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.771400    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-openshift-ca\") pod \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\" (UID: \"7723e8a2-0ece-4771-bfb3-f6b6585ee894\") "
Nov 01 10:00:13.771751 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.771721    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-openshift-ca" pod "7723e8a2-0ece-4771-bfb3-f6b6585ee894" (UID: "7723e8a2-0ece-4771-bfb3-f6b6585ee894"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:00:13.773546 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.773515    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7723e8a2-0ece-4771-bfb3-f6b6585ee894-kube-api-access-5cpvb" pod "7723e8a2-0ece-4771-bfb3-f6b6585ee894" (UID: "7723e8a2-0ece-4771-bfb3-f6b6585ee894"). InnerVolumeSpecName "kube-api-access-5cpvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:00:13.773888 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.773862    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-xks-ca" pod "7723e8a2-0ece-4771-bfb3-f6b6585ee894" (UID: "7723e8a2-0ece-4771-bfb3-f6b6585ee894"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:00:13.872919 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.872883    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-openshift-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:13.872919 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.872914    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpvb\" (UniqueName: \"kubernetes.io/projected/7723e8a2-0ece-4771-bfb3-f6b6585ee894-kube-api-access-5cpvb\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:13.872919 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:13.872925    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7723e8a2-0ece-4771-bfb3-f6b6585ee894-maas-api-xks-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:14.500127 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:14.500094    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"
Nov 01 10:00:14.500127 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:14.500119    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4" event={"ID":"7723e8a2-0ece-4771-bfb3-f6b6585ee894","Type":"ContainerDied","Data":"36ed8d451ff265db2f9b4d1b3763a7594f76e3bcf7dbef54f6cd30d2561d8dda"}
Nov 01 10:00:14.500503 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:14.500170    4931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ed8d451ff265db2f9b4d1b3763a7594f76e3bcf7dbef54f6cd30d2561d8dda"
Nov 01 10:00:16.040944 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:16.040902    4931 scope.go:122] "RemoveContainer" containerID="1c1be29be0359ddbdc23780e537f004872162a93c919eae63b6543ffa515f6c0"
Nov 01 10:00:16.515583 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:16.515552    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" event={"ID":"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15","Type":"ContainerStarted","Data":"82a1b7a6d1bab90951a4be9a9c6fbd83455da8f621b4199451b48eda48e2da41"}
Nov 01 10:00:16.628128 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:16.628018    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" podStartSLOduration=16.628004745 podStartE2EDuration="16.628004745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:00:16.625671802 +0000 UTC m=+5286.740544768" watchObservedRunningTime="2026-11-01 10:00:16.628004745 +0000 UTC m=+5286.742877710"
Nov 01 10:00:17.523285 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:17.523253    4931 generic.go:356] "Generic (PLEG): container finished" podID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerID="82a1b7a6d1bab90951a4be9a9c6fbd83455da8f621b4199451b48eda48e2da41" exitCode=60
Nov 01 10:00:17.523706 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:17.523317    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" event={"ID":"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15","Type":"ContainerDied","Data":"82a1b7a6d1bab90951a4be9a9c6fbd83455da8f621b4199451b48eda48e2da41"}
Nov 01 10:00:17.523706 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:17.523362    4931 scope.go:122] "RemoveContainer" containerID="1c1be29be0359ddbdc23780e537f004872162a93c919eae63b6543ffa515f6c0"
Nov 01 10:00:17.523706 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:17.523659    4931 scope.go:122] "RemoveContainer" containerID="82a1b7a6d1bab90951a4be9a9c6fbd83455da8f621b4199451b48eda48e2da41"
Nov 01 10:00:17.523933 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:17.523911    4931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5_odh-ai-gateway-infra(5cf3e783-2393-4ab0-9dd9-3d0fb3298e15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15"
Nov 01 10:00:17.641353 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:17.641315    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"]
Nov 01 10:00:18.674814 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:18.674786    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:18.718313 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:18.718285    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-maas-api-service-ca\") pod \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\" (UID: \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\") "
Nov 01 10:00:18.718440 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:18.718393    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-kube-api-access-xhv9t\" (UniqueName: \"kubernetes.io/projected/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-kube-api-access-xhv9t\") pod \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\" (UID: \"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15\") "
Nov 01 10:00:18.718626 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:18.718603    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-maas-api-service-ca" pod "5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" (UID: "5cf3e783-2393-4ab0-9dd9-3d0fb3298e15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:00:18.720739 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:18.720711    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-kube-api-access-xhv9t" pod "5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" (UID: "5cf3e783-2393-4ab0-9dd9-3d0fb3298e15"). InnerVolumeSpecName "kube-api-access-xhv9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:00:18.819560 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:18.819480    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhv9t\" (UniqueName: \"kubernetes.io/projected/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-kube-api-access-xhv9t\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:18.819560 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:18.819518    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15-maas-api-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:00:19.537198 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:19.537170    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"
Nov 01 10:00:19.537359 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:19.537165    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5" event={"ID":"5cf3e783-2393-4ab0-9dd9-3d0fb3298e15","Type":"ContainerDied","Data":"a7e0091dfb9436c5c0cc541d7a18f69fb9094146731e6cf5c9ee85288cd21e3a"}
Nov 01 10:00:19.537359 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:19.537316    4931 scope.go:122] "RemoveContainer" containerID="82a1b7a6d1bab90951a4be9a9c6fbd83455da8f621b4199451b48eda48e2da41"
Nov 01 10:00:19.562565 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:19.562543    4931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf3e783_2393_4ab0_9dd9_3d0fb3298e15.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 10:00:19.695792 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:19.695759    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"]
Nov 01 10:00:19.743340 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:19.743310    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfdd13-29892120-l4gc5"]
Nov 01 10:00:20.047148 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:00:20.047113    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" path="/var/lib/kubelet/pods/5cf3e783-2393-4ab0-9dd9-3d0fb3298e15/volumes"
Nov 01 10:00:31.022309 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:00:31.022270    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:01:01.023619 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:01:01.023573    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:01:28.733569 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:28.733530    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"]
Nov 01 10:01:28.734077 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:28.733740    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy" containerID="cri-o://2ba11a33d17e8d8c1d6e7a9929761a1ccd5a52e33dd47f3052f2cfddbea86964" gracePeriod=30
Nov 01 10:01:31.023250 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:01:31.023205    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:01:33.957887 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:33.957849    4931 generic.go:356] "Generic (PLEG): container finished" podID="3684f345-16f3-4f74-9453-047f70f68101" containerID="2ba11a33d17e8d8c1d6e7a9929761a1ccd5a52e33dd47f3052f2cfddbea86964" exitCode=0
Nov 01 10:01:33.958271 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:33.957887    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" event={"ID":"3684f345-16f3-4f74-9453-047f70f68101","Type":"ContainerDied","Data":"2ba11a33d17e8d8c1d6e7a9929761a1ccd5a52e33dd47f3052f2cfddbea86964"}
Nov 01 10:01:33.986120 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:33.986097    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 10:01:34.145257 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145223    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-credential-socket\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145456 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145282    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-certs\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145456 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145376    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-istio-token\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-istio-token\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145456 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145422    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-data\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145639 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145457    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3684f345-16f3-4f74-9453-047f70f68101-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684f345-16f3-4f74-9453-047f70f68101-istiod-ca-cert\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145639 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145501    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-socket\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145639 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145534    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3684f345-16f3-4f74-9453-047f70f68101-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684f345-16f3-4f74-9453-047f70f68101-istio-podinfo\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145639 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145572    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-credential-socket" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.145639 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145586    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-certs" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.145639 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145613    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-kube-api-access-dwcv7\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-kube-api-access-dwcv7\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145957 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145655    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-envoy\") pod \"3684f345-16f3-4f74-9453-047f70f68101\" (UID: \"3684f345-16f3-4f74-9453-047f70f68101\") "
Nov 01 10:01:34.145957 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145788    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-data" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.145957 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145879    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3684f345-16f3-4f74-9453-047f70f68101-istiod-ca-cert" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:01:34.146189 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145977    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.146189 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.145996    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.146189 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.146011    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.146189 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.146044    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684f345-16f3-4f74-9453-047f70f68101-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.146395 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.146336    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-socket" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.148495 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.148465    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-istio-token" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:34.150278 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.150255    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-envoy" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:01:34.150385 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.150309    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3684f345-16f3-4f74-9453-047f70f68101-istio-podinfo" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 10:01:34.152239 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.152212    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-kube-api-access-dwcv7" pod "3684f345-16f3-4f74-9453-047f70f68101" (UID: "3684f345-16f3-4f74-9453-047f70f68101"). InnerVolumeSpecName "kube-api-access-dwcv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:01:34.247415 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.247378    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwcv7\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-kube-api-access-dwcv7\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.247415 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.247406    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.247415 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.247416    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684f345-16f3-4f74-9453-047f70f68101-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.247415 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.247424    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684f345-16f3-4f74-9453-047f70f68101-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.247783 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.247433    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684f345-16f3-4f74-9453-047f70f68101-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:01:34.966510 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.966482    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"
Nov 01 10:01:34.967078 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.966480    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59" event={"ID":"3684f345-16f3-4f74-9453-047f70f68101","Type":"ContainerDied","Data":"d560a75ac21c5ff47c640bc2ed3775f512060bf2afa40533277530b5b0d4785e"}
Nov 01 10:01:34.967078 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:34.966632    4931 scope.go:122] "RemoveContainer" containerID="2ba11a33d17e8d8c1d6e7a9929761a1ccd5a52e33dd47f3052f2cfddbea86964"
Nov 01 10:01:35.177242 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:35.177203    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"]
Nov 01 10:01:35.224970 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:35.224865    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-wtj59"]
Nov 01 10:01:36.047938 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:36.047902    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3684f345-16f3-4f74-9453-047f70f68101" path="/var/lib/kubelet/pods/3684f345-16f3-4f74-9453-047f70f68101/volumes"
Nov 01 10:01:50.117111 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:01:50.117071    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"]
Nov 01 10:02:01.023108 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:02:01.023067    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:02:05.137996 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.137957    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" containerID="cri-o://7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59" gracePeriod=45
Nov 01 10:02:05.392524 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.392465    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 10:02:05.438736 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.438701    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-model-cache\") pod \"68a37272-11e6-4788-8f65-2f27b56dd9b2\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") "
Nov 01 10:02:05.438908 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.438756    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68a37272-11e6-4788-8f65-2f27b56dd9b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/68a37272-11e6-4788-8f65-2f27b56dd9b2-tls-certs\") pod \"68a37272-11e6-4788-8f65-2f27b56dd9b2\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") "
Nov 01 10:02:05.438908 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.438784    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-dshm\") pod \"68a37272-11e6-4788-8f65-2f27b56dd9b2\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") "
Nov 01 10:02:05.438908 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.438813    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-home\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-home\") pod \"68a37272-11e6-4788-8f65-2f27b56dd9b2\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") "
Nov 01 10:02:05.438908 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.438846    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-tmp-dir\") pod \"68a37272-11e6-4788-8f65-2f27b56dd9b2\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") "
Nov 01 10:02:05.439150 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.438950    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a37272-11e6-4788-8f65-2f27b56dd9b2-kube-api-access-hjbhn\" (UniqueName: \"kubernetes.io/projected/68a37272-11e6-4788-8f65-2f27b56dd9b2-kube-api-access-hjbhn\") pod \"68a37272-11e6-4788-8f65-2f27b56dd9b2\" (UID: \"68a37272-11e6-4788-8f65-2f27b56dd9b2\") "
Nov 01 10:02:05.439214 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.439168    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-home" pod "68a37272-11e6-4788-8f65-2f27b56dd9b2" (UID: "68a37272-11e6-4788-8f65-2f27b56dd9b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:05.439356 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.439319    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-tmp-dir" pod "68a37272-11e6-4788-8f65-2f27b56dd9b2" (UID: "68a37272-11e6-4788-8f65-2f27b56dd9b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:05.439356 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.439348    4931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-home\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:05.439476 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.439385    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-model-cache" pod "68a37272-11e6-4788-8f65-2f27b56dd9b2" (UID: "68a37272-11e6-4788-8f65-2f27b56dd9b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:05.441298 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.441268    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a37272-11e6-4788-8f65-2f27b56dd9b2-tls-certs" pod "68a37272-11e6-4788-8f65-2f27b56dd9b2" (UID: "68a37272-11e6-4788-8f65-2f27b56dd9b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:02:05.441395 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.441337    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a37272-11e6-4788-8f65-2f27b56dd9b2-kube-api-access-hjbhn" pod "68a37272-11e6-4788-8f65-2f27b56dd9b2" (UID: "68a37272-11e6-4788-8f65-2f27b56dd9b2"). InnerVolumeSpecName "kube-api-access-hjbhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:05.441395 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.441339    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-dshm" pod "68a37272-11e6-4788-8f65-2f27b56dd9b2" (UID: "68a37272-11e6-4788-8f65-2f27b56dd9b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:05.540457 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.540426    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjbhn\" (UniqueName: \"kubernetes.io/projected/68a37272-11e6-4788-8f65-2f27b56dd9b2-kube-api-access-hjbhn\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:05.540457 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.540455    4931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-model-cache\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:05.540661 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.540467    4931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68a37272-11e6-4788-8f65-2f27b56dd9b2-tls-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:05.540661 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.540478    4931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-dshm\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:05.540661 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:05.540487    4931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68a37272-11e6-4788-8f65-2f27b56dd9b2-tmp-dir\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:06.162985 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.162952    4931 generic.go:356] "Generic (PLEG): container finished" podID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerID="7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59" exitCode=0
Nov 01 10:02:06.163574 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.163008    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"
Nov 01 10:02:06.163574 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.163048    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" event={"ID":"68a37272-11e6-4788-8f65-2f27b56dd9b2","Type":"ContainerDied","Data":"7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59"}
Nov 01 10:02:06.163574 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.163082    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" event={"ID":"68a37272-11e6-4788-8f65-2f27b56dd9b2","Type":"ContainerDied","Data":"aa7ac6a70b986ad3c437f26497b0945b72f7c08deffe2638ef64fcd19a5616a6"}
Nov 01 10:02:06.163574 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.163102    4931 scope.go:122] "RemoveContainer" containerID="7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59"
Nov 01 10:02:06.172754 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.172736    4931 scope.go:122] "RemoveContainer" containerID="7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59"
Nov 01 10:02:06.173049 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:02:06.173002    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59\": container with ID starting with 7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59 not found: ID does not exist" containerID="7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59"
Nov 01 10:02:06.173126 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.173066    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59"} err="failed to get container status \"7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59\": rpc error: code = NotFound desc = could not find container \"7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59\": container with ID starting with 7cbdff1c59e772e318640584cd40ff03ceaae02a3d4aeb4472df4f1300129a59 not found: ID does not exist"
Nov 01 10:02:06.311170 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.311087    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 10:02:06.322786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.322755    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"]
Nov 01 10:02:06.370822 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:06.370785    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/premium-w1-dfdd13-kserve-6655b66df9-wrsnk"]
Nov 01 10:02:08.046629 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:08.046599    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" path="/var/lib/kubelet/pods/68a37272-11e6-4788-8f65-2f27b56dd9b2/volumes"
Nov 01 10:02:11.022967 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:11.022858    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:02:11.036229 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:11.026959    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:02:11.055791 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:11.055767    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:02:11.059684 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:11.059660    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:02:19.887658 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:19.887617    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"]
Nov 01 10:02:19.888143 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:19.887830    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" containerID="cri-o://1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f" gracePeriod=30
Nov 01 10:02:24.124517 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:24.124483    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-4m4gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 10:02:24.125017 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:24.124547    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 10:02:25.142411 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.142381    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 10:02:25.229896 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.229817    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-envoy\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.229896 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.229859    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-certs\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.229896 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.229886    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-credential-socket\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.230205 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.229929    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-socket\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.230205 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.229961    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-podinfo\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.230205 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230003    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-kube-api-access-rmhj9\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-kube-api-access-rmhj9\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.230205 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230055    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-token\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-token\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.230205 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230109    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-data\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.230205 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230164    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-certs" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:25.230205 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230204    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e640a6-6a90-4f20-aecb-8470965c2f0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e640a6-6a90-4f20-aecb-8470965c2f0c-istiod-ca-cert\") pod \"40e640a6-6a90-4f20-aecb-8470965c2f0c\" (UID: \"40e640a6-6a90-4f20-aecb-8470965c2f0c\") "
Nov 01 10:02:25.230578 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230251    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-credential-socket" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:25.230578 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230268    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-socket" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:25.230578 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230538    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e640a6-6a90-4f20-aecb-8470965c2f0c-istiod-ca-cert" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:02:25.230815 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230627    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.230815 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230708    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.230815 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230723    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.230815 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.230757    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-data" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:25.232420 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.232393    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-envoy" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:25.232717 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.232697    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-token" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:25.232811 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.232784    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-podinfo" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 10:02:25.232954 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.232937    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-kube-api-access-rmhj9" pod "40e640a6-6a90-4f20-aecb-8470965c2f0c" (UID: "40e640a6-6a90-4f20-aecb-8470965c2f0c"). InnerVolumeSpecName "kube-api-access-rmhj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:25.268631 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.268602    4931 generic.go:356] "Generic (PLEG): container finished" podID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerID="1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f" exitCode=0
Nov 01 10:02:25.268745 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.268641    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" event={"ID":"40e640a6-6a90-4f20-aecb-8470965c2f0c","Type":"ContainerDied","Data":"1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f"}
Nov 01 10:02:25.268745 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.268660    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"
Nov 01 10:02:25.268745 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.268695    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs" event={"ID":"40e640a6-6a90-4f20-aecb-8470965c2f0c","Type":"ContainerDied","Data":"892789fd3114b37c906e97d5ccb8aec74cf3df872688a743f7f63dd6910446a3"}
Nov 01 10:02:25.268745 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.268724    4931 scope.go:122] "RemoveContainer" containerID="1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f"
Nov 01 10:02:25.279534 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.279518    4931 scope.go:122] "RemoveContainer" containerID="1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f"
Nov 01 10:02:25.279798 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:02:25.279779    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f\": container with ID starting with 1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f not found: ID does not exist" containerID="1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f"
Nov 01 10:02:25.279866 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.279812    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f"} err="failed to get container status \"1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f\": rpc error: code = NotFound desc = could not find container \"1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f\": container with ID starting with 1da05fae86125fb32d87ccd17c5322418116f9ccc5680340c295f31d066e1c7f not found: ID does not exist"
Nov 01 10:02:25.331193 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.331159    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e640a6-6a90-4f20-aecb-8470965c2f0c-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.331193 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.331187    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.331386 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.331201    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.331386 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.331214    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhj9\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-kube-api-access-rmhj9\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.331386 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.331227    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.331386 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.331239    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e640a6-6a90-4f20-aecb-8470965c2f0c-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:25.416199 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.416153    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"]
Nov 01 10:02:25.418483 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:25.418462    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-4m4gs"]
Nov 01 10:02:26.047282 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:26.047253    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" path="/var/lib/kubelet/pods/40e640a6-6a90-4f20-aecb-8470965c2f0c/volumes"
Nov 01 10:02:31.022069 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:02:31.022001    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:02:31.918485 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:31.918455    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"]
Nov 01 10:02:31.918686 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:31.918653    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy" containerID="cri-o://38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f" gracePeriod=30
Nov 01 10:02:33.565107 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:33.565077    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6s9qr_ea75c58e-9f4a-4401-bcea-548e683abe2e/manager/0.log"
Nov 01 10:02:33.838190 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:33.838115    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zslv4_5cdcfa74-fd37-405c-8850-47dfa5b238e5/manager/1.log"
Nov 01 10:02:35.264180 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:35.264145    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vlfhp_76ef7b97-d4b7-4482-9026-98a0ffffeaf1/manager/1.log"
Nov 01 10:02:36.451708 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:36.451681    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zslv4_5cdcfa74-fd37-405c-8850-47dfa5b238e5/manager/0.log"
Nov 01 10:02:37.195193 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.195138    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 10:02:37.237222 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237191    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924263da-31d9-47f5-8b0e-b748a711033c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924263da-31d9-47f5-8b0e-b748a711033c-istiod-ca-cert\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237222 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237224    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-kube-api-access-xnjdt\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-kube-api-access-xnjdt\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237417 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237250    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-credential-socket\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237417 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237270    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-certs\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237417 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237295    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924263da-31d9-47f5-8b0e-b748a711033c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924263da-31d9-47f5-8b0e-b748a711033c-istio-podinfo\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237417 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237325    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-envoy\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237417 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237358    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-data\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237417 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237413    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-istio-token\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-istio-token\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237736 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237455    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-socket\") pod \"924263da-31d9-47f5-8b0e-b748a711033c\" (UID: \"924263da-31d9-47f5-8b0e-b748a711033c\") "
Nov 01 10:02:37.237736 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237566    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-credential-socket" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:37.237736 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237575    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-certs" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:37.237736 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237622    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924263da-31d9-47f5-8b0e-b748a711033c-istiod-ca-cert" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:02:37.237961 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237802    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-socket" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:37.237961 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237804    4931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924263da-31d9-47f5-8b0e-b748a711033c-istiod-ca-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.237961 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237849    4931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-credential-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.237961 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.237865    4931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.238278 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.238004    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-data" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:37.239737 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.239713    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-istio-token" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:37.239737 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.239725    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-kube-api-access-xnjdt" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "kube-api-access-xnjdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:02:37.240256 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.240230    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-envoy" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:02:37.240357 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.240253    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924263da-31d9-47f5-8b0e-b748a711033c-istio-podinfo" pod "924263da-31d9-47f5-8b0e-b748a711033c" (UID: "924263da-31d9-47f5-8b0e-b748a711033c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 10:02:37.338208 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338174    4931 generic.go:356] "Generic (PLEG): container finished" podID="924263da-31d9-47f5-8b0e-b748a711033c" containerID="38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f" exitCode=0
Nov 01 10:02:37.338396 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338259    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"
Nov 01 10:02:37.338396 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338268    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" event={"ID":"924263da-31d9-47f5-8b0e-b748a711033c","Type":"ContainerDied","Data":"38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f"}
Nov 01 10:02:37.338396 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338348    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnjdt\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-kube-api-access-xnjdt\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.338396 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338364    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787" event={"ID":"924263da-31d9-47f5-8b0e-b748a711033c","Type":"ContainerDied","Data":"37907cd6d416af45dc89e5afcad2fef9d49434286241341f8e2c8f9e8d6ab9a5"}
Nov 01 10:02:37.338396 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338370    4931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924263da-31d9-47f5-8b0e-b748a711033c-istio-podinfo\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.338396 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338381    4931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-envoy\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.338396 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338393    4931 scope.go:122] "RemoveContainer" containerID="38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f"
Nov 01 10:02:37.338816 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338450    4931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-istio-data\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.338816 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338543    4931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924263da-31d9-47f5-8b0e-b748a711033c-istio-token\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.338816 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.338562    4931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924263da-31d9-47f5-8b0e-b748a711033c-workload-socket\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:02:37.349486 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.349471    4931 scope.go:122] "RemoveContainer" containerID="38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f"
Nov 01 10:02:37.349743 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:02:37.349727    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f\": container with ID starting with 38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f not found: ID does not exist" containerID="38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f"
Nov 01 10:02:37.349790 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.349750    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f"} err="failed to get container status \"38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f\": rpc error: code = NotFound desc = could not find container \"38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f\": container with ID starting with 38d901608accc71ad7967bfa49a33716e617147c4caf8e0eafcd9ee4cfe3c91f not found: ID does not exist"
Nov 01 10:02:37.494484 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.494399    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vlfhp_76ef7b97-d4b7-4482-9026-98a0ffffeaf1/manager/0.log"
Nov 01 10:02:37.509783 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.509747    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"]
Nov 01 10:02:37.518347 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:37.518320    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-qx787"]
Nov 01 10:02:38.047468 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:38.047437    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924263da-31d9-47f5-8b0e-b748a711033c" path="/var/lib/kubelet/pods/924263da-31d9-47f5-8b0e-b748a711033c/volumes"
Nov 01 10:02:38.368162 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:38.368131    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-z247t_cc0775c9-babb-41b8-8db2-f958a8f69c95/manager/0.log"
Nov 01 10:02:38.863722 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:38.863687    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m97nw_c07de17d-572b-4a2c-a44c-b74048d7ec0a/manager/0.log"
Nov 01 10:02:39.340570 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:39.340469    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d89f6bbc8-d6hgb_785d692c-6c25-4f84-874a-136d9e082df1/manager/0.log"
Nov 01 10:02:39.576520 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:39.576490    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dssdt_8d282f7e-e082-4cc0-ac9b-39880957b56e/server/0.log"
Nov 01 10:02:40.080183 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:02:40.080154    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dcm8b_f5f45383-80fa-4ee1-9556-96ba0b54a988/proxy/0.log"
Nov 01 10:03:01.021779 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:03:01.021699    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:03:31.022217 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:03:31.022177    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:03:55.237199 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.237161    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-d6hgb"]
Nov 01 10:03:55.237851 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.237346    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb" podUID="785d692c-6c25-4f84-874a-136d9e082df1" containerName="manager" containerID="cri-o://9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6" gracePeriod=30
Nov 01 10:03:55.493518 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.493452    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 10:03:55.632633 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.632604    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-webhook-cert\") pod \"785d692c-6c25-4f84-874a-136d9e082df1\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") "
Nov 01 10:03:55.632832 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.632640    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-metrics-certs\") pod \"785d692c-6c25-4f84-874a-136d9e082df1\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") "
Nov 01 10:03:55.632832 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.632685    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785d692c-6c25-4f84-874a-136d9e082df1-kube-api-access-twhlg\" (UniqueName: \"kubernetes.io/projected/785d692c-6c25-4f84-874a-136d9e082df1-kube-api-access-twhlg\") pod \"785d692c-6c25-4f84-874a-136d9e082df1\" (UID: \"785d692c-6c25-4f84-874a-136d9e082df1\") "
Nov 01 10:03:55.634961 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.634926    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-webhook-cert" pod "785d692c-6c25-4f84-874a-136d9e082df1" (UID: "785d692c-6c25-4f84-874a-136d9e082df1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:03:55.635097 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.634969    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-metrics-certs" pod "785d692c-6c25-4f84-874a-136d9e082df1" (UID: "785d692c-6c25-4f84-874a-136d9e082df1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:03:55.635333 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.635312    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785d692c-6c25-4f84-874a-136d9e082df1-kube-api-access-twhlg" pod "785d692c-6c25-4f84-874a-136d9e082df1" (UID: "785d692c-6c25-4f84-874a-136d9e082df1"). InnerVolumeSpecName "kube-api-access-twhlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:03:55.733650 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.733611    4931 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-webhook-cert\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:55.733650 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.733642    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/785d692c-6c25-4f84-874a-136d9e082df1-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:55.733650 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.733655    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twhlg\" (UniqueName: \"kubernetes.io/projected/785d692c-6c25-4f84-874a-136d9e082df1-kube-api-access-twhlg\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:03:55.766384 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.766292    4931 generic.go:356] "Generic (PLEG): container finished" podID="785d692c-6c25-4f84-874a-136d9e082df1" containerID="9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6" exitCode=0
Nov 01 10:03:55.766384 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.766333    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb"
Nov 01 10:03:55.766384 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.766373    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb" event={"ID":"785d692c-6c25-4f84-874a-136d9e082df1","Type":"ContainerDied","Data":"9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6"}
Nov 01 10:03:55.766624 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.766421    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d89f6bbc8-d6hgb" event={"ID":"785d692c-6c25-4f84-874a-136d9e082df1","Type":"ContainerDied","Data":"a8a87adf7f7cbfaa4f4f109bd5eed9c773a04cd941a0d2a1c8c267ec58c648b5"}
Nov 01 10:03:55.766624 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.766443    4931 scope.go:122] "RemoveContainer" containerID="9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6"
Nov 01 10:03:55.776821 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.776798    4931 scope.go:122] "RemoveContainer" containerID="9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6"
Nov 01 10:03:55.777114 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:03:55.777098    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6\": container with ID starting with 9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6 not found: ID does not exist" containerID="9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6"
Nov 01 10:03:55.777175 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.777133    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6"} err="failed to get container status \"9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6\": rpc error: code = NotFound desc = could not find container \"9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6\": container with ID starting with 9c9385a91c0f0d827cbb6d08834d02a52152dadd42af5e4df411a5257bf7fef6 not found: ID does not exist"
Nov 01 10:03:55.925738 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.925705    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-d6hgb"]
Nov 01 10:03:55.929966 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:55.929945    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d89f6bbc8-d6hgb"]
Nov 01 10:03:56.046166 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:03:56.046089    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785d692c-6c25-4f84-874a-136d9e082df1" path="/var/lib/kubelet/pods/785d692c-6c25-4f84-874a-136d9e082df1/volumes"
Nov 01 10:04:01.032676 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:04:01.032639    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:04:31.022822 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:04:31.022728    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:04:36.002773 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.002742    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"]
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003132    4931 cpu_manager.go:414] "containerMap: removing container" podUID="7723e8a2-0ece-4771-bfb3-f6b6585ee894" containerName="revoke-keys"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003143    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7723e8a2-0ece-4771-bfb3-f6b6585ee894" containerName="revoke-keys"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003155    4931 cpu_manager.go:414] "containerMap: removing container" podUID="785d692c-6c25-4f84-874a-136d9e082df1" containerName="manager"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003160    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785d692c-6c25-4f84-874a-136d9e082df1" containerName="manager"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003168    4931 cpu_manager.go:414] "containerMap: removing container" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003173    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003190    4931 cpu_manager.go:414] "containerMap: removing container" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003194    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003202    4931 cpu_manager.go:414] "containerMap: removing container" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003207    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003219    4931 cpu_manager.go:414] "containerMap: removing container" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy"
Nov 01 10:04:36.003218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003224    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003233    4931 cpu_manager.go:414] "containerMap: removing container" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003238    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003249    4931 cpu_manager.go:414] "containerMap: removing container" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003254    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003261    4931 cpu_manager.go:414] "containerMap: removing container" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003265    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003324    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003331    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003338    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="924263da-31d9-47f5-8b0e-b748a711033c" containerName="istio-proxy"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003387    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3684f345-16f3-4f74-9453-047f70f68101" containerName="istio-proxy"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003395    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e640a6-6a90-4f20-aecb-8470965c2f0c" containerName="istio-proxy"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003403    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7723e8a2-0ece-4771-bfb3-f6b6585ee894" containerName="revoke-keys"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003409    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="785d692c-6c25-4f84-874a-136d9e082df1" containerName="manager"
Nov 01 10:04:36.003786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.003416    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a37272-11e6-4788-8f65-2f27b56dd9b2" containerName="main"
Nov 01 10:04:36.006798 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.006782    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094078 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094040    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094270 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094096    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfhs\" (UniqueName: \"kubernetes.io/projected/f1ee5b01-a994-4efd-960e-93688cdfe984-kube-api-access-bcfhs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094270 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094134    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094270 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094159    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094270 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094198    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094270 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094231    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ee5b01-a994-4efd-960e-93688cdfe984-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094270 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094257    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094681 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094320    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.094681 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.094346    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.095426 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.095402    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"]
Nov 01 10:04:36.195707 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195671    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.195707 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195708    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.195976 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195757    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.195976 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195775    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfhs\" (UniqueName: \"kubernetes.io/projected/f1ee5b01-a994-4efd-960e-93688cdfe984-kube-api-access-bcfhs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.195976 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195812    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.195976 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195833    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.195976 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195870    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.195976 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195898    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ee5b01-a994-4efd-960e-93688cdfe984-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.195976 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.195930    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.196401 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.196226    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.196401 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.196269    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.196401 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.196330    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.196570 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.196472    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.196810 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.196786    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ee5b01-a994-4efd-960e-93688cdfe984-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.198279 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.198260    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.198670 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.198650    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.255964 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.255885    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfhs\" (UniqueName: \"kubernetes.io/projected/f1ee5b01-a994-4efd-960e-93688cdfe984-kube-api-access-bcfhs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.256965 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.256939    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ee5b01-a994-4efd-960e-93688cdfe984-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-52vd7\" (UID: \"f1ee5b01-a994-4efd-960e-93688cdfe984\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.319195 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.319167    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:36.747995 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:04:36.747966    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ee5b01_a994_4efd_960e_93688cdfe984.slice/crio-3b1093bd955cb42de15023ca22d53469d92b057c52d189e74879c1621f18a2ad WatchSource:0}: Error finding container 3b1093bd955cb42de15023ca22d53469d92b057c52d189e74879c1621f18a2ad: Status 404 returned error can't find the container with id 3b1093bd955cb42de15023ca22d53469d92b057c52d189e74879c1621f18a2ad
Nov 01 10:04:36.750116 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.750093    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"]
Nov 01 10:04:36.750207 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.750145    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 10:04:36.750207 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.750204    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 10:04:36.750291 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.750232    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 10:04:36.997051 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.996930    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" event={"ID":"f1ee5b01-a994-4efd-960e-93688cdfe984","Type":"ContainerStarted","Data":"324cf322533aacca381ff993a63b3428b2b33b9bb433a46fb9ae0f6a00421b72"}
Nov 01 10:04:36.997423 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:36.997055    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" event={"ID":"f1ee5b01-a994-4efd-960e-93688cdfe984","Type":"ContainerStarted","Data":"3b1093bd955cb42de15023ca22d53469d92b057c52d189e74879c1621f18a2ad"}
Nov 01 10:04:37.114774 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:37.114728    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podStartSLOduration=2.114713559 podStartE2EDuration="2.114713559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:04:37.113568666 +0000 UTC m=+5547.228441635" watchObservedRunningTime="2026-11-01 10:04:37.114713559 +0000 UTC m=+5547.229586526"
Nov 01 10:04:37.320309 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:37.320206    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:37.322135 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:37.322115    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 10:04:37.322194 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:37.322160    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 10:04:38.319904 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:38.319873    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 10:04:38.320328 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:38.319921    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 10:04:39.319906 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:39.319852    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 10:04:39.320102 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:39.319958    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 10:04:40.320492 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:40.320459    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 10:04:40.320877 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:40.320515    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 10:04:41.319637 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:41.319599    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 10:04:41.319829 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:41.319662    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 10:04:42.319912 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:42.319881    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 10:04:42.320330 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:42.319945    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 10:04:43.319598 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:43.319565    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 10:04:43.319770 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:43.319615    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 10:04:44.320091 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:44.320059    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 01 10:04:44.320477 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:44.320109    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 01 10:04:45.904248 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:45.904219    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:45.904743 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:45.904601    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:04:45.905596 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:04:45.905576    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7"
Nov 01 10:05:01.023428 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:05:01.023378    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:05:14.050293 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:14.050257    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"]
Nov 01 10:05:14.097808 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:14.097779    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-215b4574-xg2l4"]
Nov 01 10:05:16.053206 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:16.053165    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7723e8a2-0ece-4771-bfb3-f6b6585ee894" path="/var/lib/kubelet/pods/7723e8a2-0ece-4771-bfb3-f6b6585ee894/volumes"
Nov 01 10:05:21.344578 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.344544    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"]
Nov 01 10:05:21.345018 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.345005    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf3e783-2393-4ab0-9dd9-3d0fb3298e15" containerName="cleanup"
Nov 01 10:05:21.349324 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.349304    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.410754 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410711    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4cx\" (UniqueName: \"kubernetes.io/projected/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-kube-api-access-8j4cx\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.410958 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410760    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.410958 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410822    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.410958 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410855    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.410958 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410870    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.410958 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410894    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.410958 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410917    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.410958 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410934    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.411433 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.410980    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512354 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512318    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4cx\" (UniqueName: \"kubernetes.io/projected/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-kube-api-access-8j4cx\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512354 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512374    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512354 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512401    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512354 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512432    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512811 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512458    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512811 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512490    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512811 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512510    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512811 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512533    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.512811 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.512593    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.513152 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.513132    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.513503 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.513482    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.513963 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.513935    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.514289 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.514238    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.514857 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.514835    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.516352 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.516323    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.516481 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.516380    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.549634 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.549497    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"]
Nov 01 10:05:21.581830 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.581782    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4cx\" (UniqueName: \"kubernetes.io/projected/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-kube-api-access-8j4cx\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.582337 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.582314    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c755fb-ce57-44f7-9fe5-53879a8c8d1c-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rm7b4\" (UID: \"60c755fb-ce57-44f7-9fe5-53879a8c8d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:21.661721 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:21.661677    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:22.102900 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:22.102867    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"]
Nov 01 10:05:22.103337 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:05:22.103315    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c755fb_ce57_44f7_9fe5_53879a8c8d1c.slice/crio-660a7c53620b5dc10d9ce2c3c47ab3291fb74ec13bacff36bbc46048bac14786 WatchSource:0}: Error finding container 660a7c53620b5dc10d9ce2c3c47ab3291fb74ec13bacff36bbc46048bac14786: Status 404 returned error can't find the container with id 660a7c53620b5dc10d9ce2c3c47ab3291fb74ec13bacff36bbc46048bac14786
Nov 01 10:05:22.105471 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:22.105443    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 10:05:22.105597 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:22.105504    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 10:05:22.105597 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:22.105529    4931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 10:05:22.251101 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:22.251057    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" event={"ID":"60c755fb-ce57-44f7-9fe5-53879a8c8d1c","Type":"ContainerStarted","Data":"06e4f3870d79c43bf0ea6109023edf7de92bac8414443e9f1b41a315fce29230"}
Nov 01 10:05:22.251304 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:22.251141    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" event={"ID":"60c755fb-ce57-44f7-9fe5-53879a8c8d1c","Type":"ContainerStarted","Data":"660a7c53620b5dc10d9ce2c3c47ab3291fb74ec13bacff36bbc46048bac14786"}
Nov 01 10:05:22.370458 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:22.370409    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podStartSLOduration=1.37039498 podStartE2EDuration="1.37039498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:05:22.367667211 +0000 UTC m=+5592.482540179" watchObservedRunningTime="2026-11-01 10:05:22.37039498 +0000 UTC m=+5592.485267948"
Nov 01 10:05:22.665228 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:22.665132    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:23.684051 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:23.677397    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 10:05:23.684051 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:23.677462    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 10:05:24.663495 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:24.663459    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 10:05:24.663810 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:24.663780    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 10:05:25.667053 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:25.665503    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 10:05:25.667053 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:25.665582    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 10:05:26.669050 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:26.666457    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 10:05:26.669050 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:26.666520    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 10:05:27.664053 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:27.663468    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 10:05:27.664053 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:27.663526    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 10:05:28.662923 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:28.662884    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 10:05:28.663434 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:28.662976    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 10:05:29.662487 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:29.662451    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 10:05:29.662711 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:29.662515    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 10:05:31.029408 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:05:31.029370    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:05:31.663112 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:31.663071    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:31.663327 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:31.663134    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:32.704240 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:32.704192    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:32.704681 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:32.704275    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:33.746197 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:33.746155    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:33.746691 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:33.746204    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:34.746980 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:34.746931    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:34.747597 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:34.746995    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:35.797052 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:35.790176    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:35.797052 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:35.790269    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:36.667051 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:36.666136    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:36.835441 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:36.835397    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:36.835970 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:36.835484    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:37.835930 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:37.835891    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:05:37.836679 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:37.836614    4931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:05:38.075837 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:38.075801    4931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:05:38.076970 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:05:38.076948    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4"
Nov 01 10:06:01.021858 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:06:01.021756    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:06:11.810619 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:06:11.810579    4931 scope.go:122] "RemoveContainer" containerID="2702b750d13ccf9a3f619d414db61fdc10b3eb5f72d169a355c776c131e27af2"
Nov 01 10:06:31.021814 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:06:31.021772    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:07:01.024642 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:07:01.024589    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:07:11.083884 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:07:11.083759    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:07:11.091502 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:07:11.091481    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:07:11.108999 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:07:11.108974    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:07:11.113769 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:07:11.113741    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:07:31.022272 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:07:31.022193    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:08:01.021868 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:08:01.021824    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:08:03.937720 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:03.937681    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"]
Nov 01 10:08:03.942422 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:03.942395    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:03.992764 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:03.992733    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-029e39-serving-cert\""
Nov 01 10:08:04.029210 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.029179    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13eb06b9-a8d0-426f-938c-fbe3519e6354-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.029356 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.029218    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-metrics-certs\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.029356 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.029284    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.029356 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.029343    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gfs\" (UniqueName: \"kubernetes.io/projected/13eb06b9-a8d0-426f-938c-fbe3519e6354-kube-api-access-d6gfs\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.046981 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.046954    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"]
Nov 01 10:08:04.130677 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.130650    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13eb06b9-a8d0-426f-938c-fbe3519e6354-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.130868 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.130682    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-metrics-certs\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.130868 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.130702    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.130868 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.130722    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gfs\" (UniqueName: \"kubernetes.io/projected/13eb06b9-a8d0-426f-938c-fbe3519e6354-kube-api-access-d6gfs\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.130868 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:08:04.130825    4931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-serving-cert: secret "maas-api-e2e-worker-single-main-029e39-serving-cert" not found
Nov 01 10:08:04.131157 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:08:04.130898    4931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls podName:13eb06b9-a8d0-426f-938c-fbe3519e6354 nodeName:}" failed. No retries permitted until 2026-11-01 10:08:04.630881042 +0000 UTC m=+5754.745754007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls") pod "maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2" (UID: "13eb06b9-a8d0-426f-938c-fbe3519e6354") : secret "maas-api-e2e-worker-single-main-029e39-serving-cert" not found
Nov 01 10:08:04.131419 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.131397    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13eb06b9-a8d0-426f-938c-fbe3519e6354-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.133169 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.133147    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-metrics-certs\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.190117 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.190055    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gfs\" (UniqueName: \"kubernetes.io/projected/13eb06b9-a8d0-426f-938c-fbe3519e6354-kube-api-access-d6gfs\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.636355 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.636326    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.638891 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.638870    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:04.854854 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:04.854825    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:05.274707 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:05.274679    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"]
Nov 01 10:08:05.277672 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:05.277653    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 10:08:05.385486 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:05.385455    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2" event={"ID":"13eb06b9-a8d0-426f-938c-fbe3519e6354","Type":"ContainerStarted","Data":"903ecdc04e6314730d66ad23049f8d5aea566fd8c5f0b21028e1ab79ba339d81"}
Nov 01 10:08:06.392120 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:06.392082    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2" event={"ID":"13eb06b9-a8d0-426f-938c-fbe3519e6354","Type":"ContainerStarted","Data":"051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c"}
Nov 01 10:08:06.392556 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:06.392208    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:06.508445 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:06.508344    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2" podStartSLOduration=2.5925303509999997 podStartE2EDuration="3.50832991s" totalImagesPullingTime="915.799559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:08:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:08:06.506071589 +0000 UTC m=+5756.620944557" watchObservedRunningTime="2026-11-01 10:08:06.50832991 +0000 UTC m=+5756.623202876"
Nov 01 10:08:12.403678 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:08:12.403647    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:08:31.023844 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:08:31.023807    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:09:01.022299 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:09:01.022219    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:09:31.022402 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:09:31.022360    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:09:36.026628 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:36.026583    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"]
Nov 01 10:09:36.027336 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:36.026792    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2" podUID="13eb06b9-a8d0-426f-938c-fbe3519e6354" containerName="maas-api" containerID="cri-o://051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c" gracePeriod=30
Nov 01 10:09:37.297669 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.297643    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:09:37.319792 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.319757    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13eb06b9-a8d0-426f-938c-fbe3519e6354-kube-api-access-d6gfs\" (UniqueName: \"kubernetes.io/projected/13eb06b9-a8d0-426f-938c-fbe3519e6354-kube-api-access-d6gfs\") pod \"13eb06b9-a8d0-426f-938c-fbe3519e6354\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") "
Nov 01 10:09:37.319954 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.319824    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls\") pod \"13eb06b9-a8d0-426f-938c-fbe3519e6354\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") "
Nov 01 10:09:37.319954 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.319852    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-metrics-certs\") pod \"13eb06b9-a8d0-426f-938c-fbe3519e6354\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") "
Nov 01 10:09:37.319954 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.319941    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13eb06b9-a8d0-426f-938c-fbe3519e6354-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13eb06b9-a8d0-426f-938c-fbe3519e6354-openshift-service-ca\") pod \"13eb06b9-a8d0-426f-938c-fbe3519e6354\" (UID: \"13eb06b9-a8d0-426f-938c-fbe3519e6354\") "
Nov 01 10:09:37.320373 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.320346    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13eb06b9-a8d0-426f-938c-fbe3519e6354-openshift-service-ca" pod "13eb06b9-a8d0-426f-938c-fbe3519e6354" (UID: "13eb06b9-a8d0-426f-938c-fbe3519e6354"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:09:37.322086 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.322059    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13eb06b9-a8d0-426f-938c-fbe3519e6354-kube-api-access-d6gfs" pod "13eb06b9-a8d0-426f-938c-fbe3519e6354" (UID: "13eb06b9-a8d0-426f-938c-fbe3519e6354"). InnerVolumeSpecName "kube-api-access-d6gfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:09:37.322630 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.322609    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls" pod "13eb06b9-a8d0-426f-938c-fbe3519e6354" (UID: "13eb06b9-a8d0-426f-938c-fbe3519e6354"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:09:37.322688 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.322622    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-metrics-certs" pod "13eb06b9-a8d0-426f-938c-fbe3519e6354" (UID: "13eb06b9-a8d0-426f-938c-fbe3519e6354"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:09:37.421579 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.421500    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13eb06b9-a8d0-426f-938c-fbe3519e6354-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:09:37.421579 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.421539    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6gfs\" (UniqueName: \"kubernetes.io/projected/13eb06b9-a8d0-426f-938c-fbe3519e6354-kube-api-access-d6gfs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:09:37.421579 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.421557    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:09:37.421579 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.421571    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13eb06b9-a8d0-426f-938c-fbe3519e6354-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:09:37.902280 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.902245    4931 generic.go:356] "Generic (PLEG): container finished" podID="13eb06b9-a8d0-426f-938c-fbe3519e6354" containerID="051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c" exitCode=0
Nov 01 10:09:37.902468 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.902311    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2" event={"ID":"13eb06b9-a8d0-426f-938c-fbe3519e6354","Type":"ContainerDied","Data":"051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c"}
Nov 01 10:09:37.902468 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.902357    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2" event={"ID":"13eb06b9-a8d0-426f-938c-fbe3519e6354","Type":"ContainerDied","Data":"903ecdc04e6314730d66ad23049f8d5aea566fd8c5f0b21028e1ab79ba339d81"}
Nov 01 10:09:37.902468 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.902378    4931 scope.go:122] "RemoveContainer" containerID="051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c"
Nov 01 10:09:37.902468 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.902297    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"
Nov 01 10:09:37.912265 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.912245    4931 scope.go:122] "RemoveContainer" containerID="051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c"
Nov 01 10:09:37.912599 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:09:37.912578    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c\": container with ID starting with 051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c not found: ID does not exist" containerID="051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c"
Nov 01 10:09:37.912649 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:37.912611    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c"} err="failed to get container status \"051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c\": rpc error: code = NotFound desc = could not find container \"051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c\": container with ID starting with 051292e49595d1fc7d0262e7e2b795c89530730ab434ddb960eab3b9b921867c not found: ID does not exist"
Nov 01 10:09:38.061989 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:38.061954    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"]
Nov 01 10:09:38.110100 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:38.110069    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-029e39-76d5d5894b-45sz2"]
Nov 01 10:09:40.047090 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:09:40.047051    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13eb06b9-a8d0-426f-938c-fbe3519e6354" path="/var/lib/kubelet/pods/13eb06b9-a8d0-426f-938c-fbe3519e6354/volumes"
Nov 01 10:10:01.023337 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:10:01.023297    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:10:31.031857 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:10:31.031758    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:11:01.052457 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:11:01.052418    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:11:01.948688 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:11:01.948177    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:11:01.948688 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:11:01.948249    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:11:31.058106 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:11:31.058070    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:11:31.950735 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:11:31.950214    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:11:31.950735 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:11:31.950276    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:12:01.022265 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:12:01.022187    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:12:09.124049 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:12:09.123147    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:12:09.124049 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:12:09.123250    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:12:11.128069 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:12:11.128038    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:12:11.132260 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:12:11.132236    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:12:11.153590 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:12:11.153563    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:12:11.157400 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:12:11.157379    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:12:16.951801 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:12:16.951171    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:12:16.951801 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:12:16.951261    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:12:31.021831 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:12:31.021793    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:13:01.021217 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:13:01.021174    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:13:31.022318 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:13:31.022238    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:14:01.022828 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:14:01.022790    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:14:31.021664 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:14:31.021630    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:15:01.022250 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:15:01.022160    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:15:31.021605 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:15:31.021567    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:15:46.959224 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:15:46.959184    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:15:46.959994 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:15:46.959961    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:16:01.022785 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:16:01.022745    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:16:31.022289 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:16:31.022207    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:17:01.021342 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:17:01.021301    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:17:11.163197 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:17:11.163091    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:17:11.167258 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:17:11.167237    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:17:11.190525 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:17:11.190506    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:17:11.194104 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:17:11.194087    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:17:31.022340 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:17:31.022300    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:18:01.021765 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:18:01.021676    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:18:09.126268 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:18:09.126206    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:18:09.126929 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:18:09.126315    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:18:31.029427 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:18:31.029388    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:19:01.022569 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:19:01.022530    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:19:31.021743 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:19:31.021654    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:20:01.022298 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:20:01.022255    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:20:16.953411 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:20:16.953368    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:20:16.954004 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:20:16.953461    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:20:31.021827 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:20:31.021787    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:21:01.022610 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:21:01.022522    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:21:31.022277 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:21:31.022238    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:22:01.025214 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:22:01.025170    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:22:11.198424 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:22:11.198319    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:22:11.202235 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:22:11.202210    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:22:11.234512 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:22:11.234489    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:22:11.238193 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:22:11.238176    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:22:31.022757 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:22:31.022668    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:22:59.363495 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:22:59.363461    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"]
Nov 01 10:22:59.364107 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:22:59.363654    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp" podUID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerName="manager" containerID="cri-o://c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9" gracePeriod=10
Nov 01 10:23:00.423387 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.423358    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 10:23:00.543653 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.543571    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-extensions-socket-volume\") pod \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\" (UID: \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\") "
Nov 01 10:23:00.543804 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.543710    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-kube-api-access-thw78\" (UniqueName: \"kubernetes.io/projected/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-kube-api-access-thw78\") pod \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\" (UID: \"76ef7b97-d4b7-4482-9026-98a0ffffeaf1\") "
Nov 01 10:23:00.543955 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.543932    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-extensions-socket-volume" pod "76ef7b97-d4b7-4482-9026-98a0ffffeaf1" (UID: "76ef7b97-d4b7-4482-9026-98a0ffffeaf1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 10:23:00.545923 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.545894    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-kube-api-access-thw78" pod "76ef7b97-d4b7-4482-9026-98a0ffffeaf1" (UID: "76ef7b97-d4b7-4482-9026-98a0ffffeaf1"). InnerVolumeSpecName "kube-api-access-thw78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:23:00.645557 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.645503    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thw78\" (UniqueName: \"kubernetes.io/projected/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-kube-api-access-thw78\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:23:00.645557 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.645548    4931 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76ef7b97-d4b7-4482-9026-98a0ffffeaf1-extensions-socket-volume\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:23:00.774125 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.774094    4931 generic.go:356] "Generic (PLEG): container finished" podID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerID="c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9" exitCode=0
Nov 01 10:23:00.774309 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.774141    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"
Nov 01 10:23:00.774309 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.774186    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp" event={"ID":"76ef7b97-d4b7-4482-9026-98a0ffffeaf1","Type":"ContainerDied","Data":"c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9"}
Nov 01 10:23:00.774309 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.774243    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp" event={"ID":"76ef7b97-d4b7-4482-9026-98a0ffffeaf1","Type":"ContainerDied","Data":"eb1cdbdaffc22dabc1e4c2bbeec7951406fec9a93fd2a6caf9c386050b47dde1"}
Nov 01 10:23:00.774309 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.774271    4931 scope.go:122] "RemoveContainer" containerID="c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9"
Nov 01 10:23:00.787241 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.787221    4931 scope.go:122] "RemoveContainer" containerID="3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a"
Nov 01 10:23:00.799239 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.799221    4931 scope.go:122] "RemoveContainer" containerID="c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9"
Nov 01 10:23:00.799501 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:23:00.799483    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9\": container with ID starting with c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9 not found: ID does not exist" containerID="c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9"
Nov 01 10:23:00.799579 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.799515    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9"} err="failed to get container status \"c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9\": rpc error: code = NotFound desc = could not find container \"c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9\": container with ID starting with c9e0800d83f34ac181cfd320ed2d2fe4ae8c3640fcedb06f92e7109bbc70e9d9 not found: ID does not exist"
Nov 01 10:23:00.799579 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.799541    4931 scope.go:122] "RemoveContainer" containerID="3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a"
Nov 01 10:23:00.799769 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:23:00.799753    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a\": container with ID starting with 3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a not found: ID does not exist" containerID="3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a"
Nov 01 10:23:00.799823 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.799777    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a"} err="failed to get container status \"3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a\": rpc error: code = NotFound desc = could not find container \"3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a\": container with ID starting with 3db0b4a6915bcb6afdeba1c465125d74b5785ff058bfe189b006c5112df1304a not found: ID does not exist"
Nov 01 10:23:00.933812 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.933778    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"]
Nov 01 10:23:00.981548 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:00.981516    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vlfhp"]
Nov 01 10:23:01.022524 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:23:01.022491    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:23:02.047071 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:23:02.047011    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" path="/var/lib/kubelet/pods/76ef7b97-d4b7-4482-9026-98a0ffffeaf1/volumes"
Nov 01 10:23:31.021928 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:23:31.021884    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:24:01.023327 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:24:01.023240    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:24:02.955780 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.955746    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"]
Nov 01 10:24:02.956222 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956151    4931 cpu_manager.go:414] "containerMap: removing container" podUID="13eb06b9-a8d0-426f-938c-fbe3519e6354" containerName="maas-api"
Nov 01 10:24:02.956222 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956162    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13eb06b9-a8d0-426f-938c-fbe3519e6354" containerName="maas-api"
Nov 01 10:24:02.956222 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956179    4931 cpu_manager.go:414] "containerMap: removing container" podUID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerName="manager"
Nov 01 10:24:02.956222 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956185    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerName="manager"
Nov 01 10:24:02.956222 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956195    4931 cpu_manager.go:414] "containerMap: removing container" podUID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerName="manager"
Nov 01 10:24:02.956222 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956200    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerName="manager"
Nov 01 10:24:02.956415 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956285    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerName="manager"
Nov 01 10:24:02.956415 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956295    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ef7b97-d4b7-4482-9026-98a0ffffeaf1" containerName="manager"
Nov 01 10:24:02.956415 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.956307    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="13eb06b9-a8d0-426f-938c-fbe3519e6354" containerName="maas-api"
Nov 01 10:24:02.959429 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:02.959414    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:03.009635 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.009600    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"]
Nov 01 10:24:03.055345 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.055312    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2vkbl\""
Nov 01 10:24:03.115488 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.115453    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e7e508b-b94f-4e2a-9bf1-c994ed627324-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-svjt2\" (UID: \"2e7e508b-b94f-4e2a-9bf1-c994ed627324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:03.115664 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.115512    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ckhn\" (UniqueName: \"kubernetes.io/projected/2e7e508b-b94f-4e2a-9bf1-c994ed627324-kube-api-access-8ckhn\") pod \"kuadrant-operator-controller-manager-678d87766f-svjt2\" (UID: \"2e7e508b-b94f-4e2a-9bf1-c994ed627324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:03.216278 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.216186    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e7e508b-b94f-4e2a-9bf1-c994ed627324-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-svjt2\" (UID: \"2e7e508b-b94f-4e2a-9bf1-c994ed627324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:03.216278 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.216244    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ckhn\" (UniqueName: \"kubernetes.io/projected/2e7e508b-b94f-4e2a-9bf1-c994ed627324-kube-api-access-8ckhn\") pod \"kuadrant-operator-controller-manager-678d87766f-svjt2\" (UID: \"2e7e508b-b94f-4e2a-9bf1-c994ed627324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:03.216597 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.216577    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e7e508b-b94f-4e2a-9bf1-c994ed627324-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-svjt2\" (UID: \"2e7e508b-b94f-4e2a-9bf1-c994ed627324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:03.275374 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.275339    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ckhn\" (UniqueName: \"kubernetes.io/projected/2e7e508b-b94f-4e2a-9bf1-c994ed627324-kube-api-access-8ckhn\") pod \"kuadrant-operator-controller-manager-678d87766f-svjt2\" (UID: \"2e7e508b-b94f-4e2a-9bf1-c994ed627324\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:03.572772 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:03.572683    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:04.037581 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:04.037554    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"]
Nov 01 10:24:04.039005 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:24:04.038983    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7e508b_b94f_4e2a_9bf1_c994ed627324.slice/crio-3395ef7e6a5553322e856ce42b65febac776e15fd91455b567144b8492891eb0 WatchSource:0}: Error finding container 3395ef7e6a5553322e856ce42b65febac776e15fd91455b567144b8492891eb0: Status 404 returned error can't find the container with id 3395ef7e6a5553322e856ce42b65febac776e15fd91455b567144b8492891eb0
Nov 01 10:24:04.106069 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:04.106009    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2" event={"ID":"2e7e508b-b94f-4e2a-9bf1-c994ed627324","Type":"ContainerStarted","Data":"36591959565fb9067f0caffa80a988b180b6be71660e86cf122920dee70dd495"}
Nov 01 10:24:04.106218 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:04.106088    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2" event={"ID":"2e7e508b-b94f-4e2a-9bf1-c994ed627324","Type":"ContainerStarted","Data":"3395ef7e6a5553322e856ce42b65febac776e15fd91455b567144b8492891eb0"}
Nov 01 10:24:04.270546 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:04.270477    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2" podStartSLOduration=2.270462217 podStartE2EDuration="2.270462217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:24:04.222708504 +0000 UTC m=+6714.337581484" watchObservedRunningTime="2026-11-01 10:24:04.270462217 +0000 UTC m=+6714.385335184"
Nov 01 10:24:05.111415 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:05.111375    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:16.120695 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:16.120662    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-svjt2"
Nov 01 10:24:31.021506 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:24:31.021463    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:24:46.947223 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:46.947181    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:24:46.947744 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:46.947255    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:24:54.125231 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:54.124119    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:24:54.125231 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:24:54.124197    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:25:01.022323 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:25:01.022273    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:25:14.037771 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.037737    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"]
Nov 01 10:25:14.041699 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.041674    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.142109 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.142075    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"]
Nov 01 10:25:14.142421 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.142402    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91556f89-d14e-4c21-8985-51fddab890f0-openshift-service-ca\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.142533 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.142508    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91556f89-d14e-4c21-8985-51fddab890f0-metrics-certs\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.142679 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.142547    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91556f89-d14e-4c21-8985-51fddab890f0-maas-api-tls\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.142764 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.142737    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjv59\" (UniqueName: \"kubernetes.io/projected/91556f89-d14e-4c21-8985-51fddab890f0-kube-api-access-kjv59\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.243668 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.243626    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjv59\" (UniqueName: \"kubernetes.io/projected/91556f89-d14e-4c21-8985-51fddab890f0-kube-api-access-kjv59\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.243875 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.243681    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91556f89-d14e-4c21-8985-51fddab890f0-openshift-service-ca\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.243875 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.243731    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91556f89-d14e-4c21-8985-51fddab890f0-metrics-certs\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.243875 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.243756    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91556f89-d14e-4c21-8985-51fddab890f0-maas-api-tls\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.244420 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.244395    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91556f89-d14e-4c21-8985-51fddab890f0-openshift-service-ca\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.246434 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.246413    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91556f89-d14e-4c21-8985-51fddab890f0-metrics-certs\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.246518 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.246448    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91556f89-d14e-4c21-8985-51fddab890f0-maas-api-tls\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.303285 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.303225    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjv59\" (UniqueName: \"kubernetes.io/projected/91556f89-d14e-4c21-8985-51fddab890f0-kube-api-access-kjv59\") pod \"maas-api-676b7bff79-wzg7z\" (UID: \"91556f89-d14e-4c21-8985-51fddab890f0\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.355422 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.355388    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:14.731733 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.731707    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"]
Nov 01 10:25:14.733727 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:25:14.733698    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91556f89_d14e_4c21_8985_51fddab890f0.slice/crio-ec491c2d7c3d72d3b1138a1e95479f1a01d71973a5b3424f55c0e047829a35be WatchSource:0}: Error finding container ec491c2d7c3d72d3b1138a1e95479f1a01d71973a5b3424f55c0e047829a35be: Status 404 returned error can't find the container with id ec491c2d7c3d72d3b1138a1e95479f1a01d71973a5b3424f55c0e047829a35be
Nov 01 10:25:14.735047 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:14.735013    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 10:25:15.547871 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:15.547837    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z" event={"ID":"91556f89-d14e-4c21-8985-51fddab890f0","Type":"ContainerStarted","Data":"ec491c2d7c3d72d3b1138a1e95479f1a01d71973a5b3424f55c0e047829a35be"}
Nov 01 10:25:16.554824 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:16.554787    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z" event={"ID":"91556f89-d14e-4c21-8985-51fddab890f0","Type":"ContainerStarted","Data":"8e349174e4d861c88e5ebc0e12c0aadc39cf5a7cdb445e58f0a76d8e6655ddcf"}
Nov 01 10:25:16.555235 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:16.554921    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:16.674380 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:16.674333    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z" podStartSLOduration=2.73090543 podStartE2EDuration="3.674320769s" totalImagesPullingTime="943.415339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:25:16.674251831 +0000 UTC m=+6786.789124812" watchObservedRunningTime="2026-11-01 10:25:16.674320769 +0000 UTC m=+6786.789193762"
Nov 01 10:25:22.566242 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:22.566163    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-676b7bff79-wzg7z"
Nov 01 10:25:22.732240 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:22.732200    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"]
Nov 01 10:25:22.732501 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:22.732473    4931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv" podUID="034aef14-4a57-4833-bfb8-de72c3882304" containerName="maas-api" containerID="cri-o://1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb" gracePeriod=30
Nov 01 10:25:22.764660 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:22.764625    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv" podUID="034aef14-4a57-4833-bfb8-de72c3882304" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.52:8443/health\": dial tcp 10.133.0.52:8443: connect: connection refused"
Nov 01 10:25:24.034382 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.034348    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 10:25:24.076225 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.076133    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-metrics-certs\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-metrics-certs\") pod \"034aef14-4a57-4833-bfb8-de72c3882304\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") "
Nov 01 10:25:24.076225 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.076183    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034aef14-4a57-4833-bfb8-de72c3882304-kube-api-access-gsgxx\" (UniqueName: \"kubernetes.io/projected/034aef14-4a57-4833-bfb8-de72c3882304-kube-api-access-gsgxx\") pod \"034aef14-4a57-4833-bfb8-de72c3882304\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") "
Nov 01 10:25:24.076451 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.076271    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034aef14-4a57-4833-bfb8-de72c3882304-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034aef14-4a57-4833-bfb8-de72c3882304-openshift-service-ca\") pod \"034aef14-4a57-4833-bfb8-de72c3882304\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") "
Nov 01 10:25:24.076451 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.076312    4931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-maas-api-tls\") pod \"034aef14-4a57-4833-bfb8-de72c3882304\" (UID: \"034aef14-4a57-4833-bfb8-de72c3882304\") "
Nov 01 10:25:24.076773 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.076729    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034aef14-4a57-4833-bfb8-de72c3882304-openshift-service-ca" pod "034aef14-4a57-4833-bfb8-de72c3882304" (UID: "034aef14-4a57-4833-bfb8-de72c3882304"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 10:25:24.078773 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.078748    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-maas-api-tls" pod "034aef14-4a57-4833-bfb8-de72c3882304" (UID: "034aef14-4a57-4833-bfb8-de72c3882304"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:25:24.079203 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.079179    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-metrics-certs" pod "034aef14-4a57-4833-bfb8-de72c3882304" (UID: "034aef14-4a57-4833-bfb8-de72c3882304"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 10:25:24.081206 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.081185    4931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034aef14-4a57-4833-bfb8-de72c3882304-kube-api-access-gsgxx" pod "034aef14-4a57-4833-bfb8-de72c3882304" (UID: "034aef14-4a57-4833-bfb8-de72c3882304"). InnerVolumeSpecName "kube-api-access-gsgxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 10:25:24.178271 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.178233    4931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034aef14-4a57-4833-bfb8-de72c3882304-openshift-service-ca\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:24.178271 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.178273    4931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-maas-api-tls\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:24.178479 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.178290    4931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034aef14-4a57-4833-bfb8-de72c3882304-metrics-certs\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:24.178479 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.178304    4931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsgxx\" (UniqueName: \"kubernetes.io/projected/034aef14-4a57-4833-bfb8-de72c3882304-kube-api-access-gsgxx\") on node \"ip-10-0-143-244.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 10:25:24.635049 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.634876    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv" event={"ID":"034aef14-4a57-4833-bfb8-de72c3882304","Type":"ContainerDied","Data":"1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb"}
Nov 01 10:25:24.635049 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.634917    4931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"
Nov 01 10:25:24.635049 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.634957    4931 scope.go:122] "RemoveContainer" containerID="1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb"
Nov 01 10:25:24.635411 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.634831    4931 generic.go:356] "Generic (PLEG): container finished" podID="034aef14-4a57-4833-bfb8-de72c3882304" containerID="1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb" exitCode=0
Nov 01 10:25:24.635477 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.635427    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv" event={"ID":"034aef14-4a57-4833-bfb8-de72c3882304","Type":"ContainerDied","Data":"20a827522baa064fb4b88f22408e600a0fff466b2ea7166f60ec2517b6ada6eb"}
Nov 01 10:25:24.646801 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.646783    4931 scope.go:122] "RemoveContainer" containerID="1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb"
Nov 01 10:25:24.647087 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:25:24.647058    4931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb\": container with ID starting with 1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb not found: ID does not exist" containerID="1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb"
Nov 01 10:25:24.647177 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.647096    4931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb"} err="failed to get container status \"1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb\": rpc error: code = NotFound desc = could not find container \"1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb\": container with ID starting with 1196884f6d88b4435245905cda183b9d3d0c83713ae61e26cd90c271ae28ecdb not found: ID does not exist"
Nov 01 10:25:24.810050 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.809046    4931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"]
Nov 01 10:25:24.855489 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:24.855451    4931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-dm9cv"]
Nov 01 10:25:26.047074 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:25:26.047015    4931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034aef14-4a57-4833-bfb8-de72c3882304" path="/var/lib/kubelet/pods/034aef14-4a57-4833-bfb8-de72c3882304/volumes"
Nov 01 10:25:31.021945 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:25:31.021907    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:26:01.021957 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:26:01.021917    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:26:31.021832 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:26:31.021790    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:26:46.971159 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:26:46.971121    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:26:46.971897 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:26:46.971862    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:27:01.022581 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:27:01.022499    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:27:11.234767 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:27:11.234655    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:27:11.238752 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:27:11.238477    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:27:11.271301 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:27:11.271272    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:27:11.275215 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:27:11.275197    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:27:31.021778 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:27:31.021736    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:28:01.022327 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:28:01.022289    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:28:25.220120 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:28:25.219120    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:28:25.220120 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:28:25.219218    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:28:31.143506 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:28:31.143450    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:28:31.951212 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:28:31.951176    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:28:31.951430 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:28:31.951247    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:28:55.231049 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:28:55.226426    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:28:55.231049 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:28:55.226483    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:29:01.021903 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:29:01.021863    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:29:31.021931 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:29:31.021893    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:29:39.125050 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:29:39.124133    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rm7b4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:29:39.125050 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:29:39.124212    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rm7b4" podUID="60c755fb-ce57-44f7-9fe5-53879a8c8d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:29:40.215178 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:29:40.215137    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:29:40.216774 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:29:40.216737    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:30:01.061257 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:30:01.061132    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:30:01.955180 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:30:01.955137    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-52vd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:30:01.955360 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:30:01.955205    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-52vd7" podUID="f1ee5b01-a994-4efd-960e-93688cdfe984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:30:31.021849 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:30:31.021806    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:30:55.218217 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:30:55.218179    4931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-5gxgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 10:30:55.218991 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:30:55.218965    4931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-5gxgq" podUID="ca5beb2f-1ccf-40e5-a862-e0264dbd35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 10:31:01.021989 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:31:01.021953    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:31:31.021849 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:31:31.021813    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:31:40.977885 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:40.977858    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-z247t_cc0775c9-babb-41b8-8db2-f958a8f69c95/manager/0.log"
Nov 01 10:31:41.483641 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:41.483599    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m97nw_c07de17d-572b-4a2c-a44c-b74048d7ec0a/manager/0.log"
Nov 01 10:31:42.076460 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:42.076436    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dssdt_8d282f7e-e082-4cc0-ac9b-39880957b56e/server/0.log"
Nov 01 10:31:42.571794 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:42.571763    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dcm8b_f5f45383-80fa-4ee1-9556-96ba0b54a988/proxy/0.log"
Nov 01 10:31:43.794915 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:43.794874    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-676b7bff79-wzg7z_91556f89-d14e-4c21-8985-51fddab890f0/maas-api/0.log"
Nov 01 10:31:47.545712 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:47.545681    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bpdpw_689189ee-73d6-41dc-920f-5f3108439b62/postgres/0.log"
Nov 01 10:31:49.913620 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:49.913590    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6s9qr_ea75c58e-9f4a-4401-bcea-548e683abe2e/manager/0.log"
Nov 01 10:31:50.154504 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:50.154475    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zslv4_5cdcfa74-fd37-405c-8850-47dfa5b238e5/manager/1.log"
Nov 01 10:31:50.618926 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:50.618894    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-svjt2_2e7e508b-b94f-4e2a-9bf1-c994ed627324/manager/0.log"
Nov 01 10:31:51.830956 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:51.830919    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zslv4_5cdcfa74-fd37-405c-8850-47dfa5b238e5/manager/0.log"
Nov 01 10:31:52.401560 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:52.401517    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-557c4f746fqdwtg_75d1c892-bf07-467c-92db-655a8d317413/istio-proxy/0.log"
Nov 01 10:31:53.349845 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:53.349817    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-52vd7_f1ee5b01-a994-4efd-960e-93688cdfe984/istio-proxy/0.log"
Nov 01 10:31:53.590344 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:53.590314    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-5gxgq_ca5beb2f-1ccf-40e5-a862-e0264dbd35e4/istio-proxy/0.log"
Nov 01 10:31:55.219239 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:55.219211    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-rm7b4_60c755fb-ce57-44f7-9fe5-53879a8c8d1c/istio-proxy/0.log"
Nov 01 10:31:55.686697 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:55.686667    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65_0b60d387-603f-4171-ac01-7a17b3dfddad/istio-proxy/0.log"
Nov 01 10:31:56.356039 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:31:56.355998    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6bf599bf-9cjkf_3ca1667a-32ed-4777-8755-ec7e72add722/router/0.log"
Nov 01 10:32:01.021934 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:32:01.021893    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:32:08.900834 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:08.900798    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w96kq/must-gather-jdbxx"]
Nov 01 10:32:08.901266 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:08.901255    4931 cpu_manager.go:414] "containerMap: removing container" podUID="034aef14-4a57-4833-bfb8-de72c3882304" containerName="maas-api"
Nov 01 10:32:08.901312 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:08.901269    4931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034aef14-4a57-4833-bfb8-de72c3882304" containerName="maas-api"
Nov 01 10:32:08.901371 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:08.901360    4931 memory_manager.go:365] "RemoveStaleState removing state" podUID="034aef14-4a57-4833-bfb8-de72c3882304" containerName="maas-api"
Nov 01 10:32:08.904634 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:08.904618    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w96kq/must-gather-jdbxx"
Nov 01 10:32:08.930844 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:08.930815    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb-must-gather-output\") pod \"must-gather-jdbxx\" (UID: \"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb\") " pod="openshift-must-gather-w96kq/must-gather-jdbxx"
Nov 01 10:32:08.930961 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:08.930880    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mmw6\" (UniqueName: \"kubernetes.io/projected/9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb-kube-api-access-4mmw6\") pod \"must-gather-jdbxx\" (UID: \"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb\") " pod="openshift-must-gather-w96kq/must-gather-jdbxx"
Nov 01 10:32:09.004101 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.004062    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w96kq\"/\"kube-root-ca.crt\""
Nov 01 10:32:09.004101 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.004062    4931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w96kq\"/\"openshift-service-ca.crt\""
Nov 01 10:32:09.004317 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.004145    4931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w96kq\"/\"default-dockercfg-f9ll4\""
Nov 01 10:32:09.006311 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.006282    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w96kq/must-gather-jdbxx"]
Nov 01 10:32:09.032167 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.032133    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mmw6\" (UniqueName: \"kubernetes.io/projected/9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb-kube-api-access-4mmw6\") pod \"must-gather-jdbxx\" (UID: \"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb\") " pod="openshift-must-gather-w96kq/must-gather-jdbxx"
Nov 01 10:32:09.032361 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.032223    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb-must-gather-output\") pod \"must-gather-jdbxx\" (UID: \"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb\") " pod="openshift-must-gather-w96kq/must-gather-jdbxx"
Nov 01 10:32:09.032621 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.032602    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb-must-gather-output\") pod \"must-gather-jdbxx\" (UID: \"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb\") " pod="openshift-must-gather-w96kq/must-gather-jdbxx"
Nov 01 10:32:09.091117 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.091081    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mmw6\" (UniqueName: \"kubernetes.io/projected/9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb-kube-api-access-4mmw6\") pod \"must-gather-jdbxx\" (UID: \"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb\") " pod="openshift-must-gather-w96kq/must-gather-jdbxx"
Nov 01 10:32:09.219215 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.219143    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w96kq/must-gather-jdbxx"
Nov 01 10:32:09.593746 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.593711    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w96kq/must-gather-jdbxx"]
Nov 01 10:32:09.594678 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:32:09.594650    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ac5e364_d5b7_4d18_bd91_ac238c0fd0bb.slice/crio-e805dd1b7937dbd1d8c850cd27fa96b24e57ed168611d641a829cdefa2e7f690 WatchSource:0}: Error finding container e805dd1b7937dbd1d8c850cd27fa96b24e57ed168611d641a829cdefa2e7f690: Status 404 returned error can't find the container with id e805dd1b7937dbd1d8c850cd27fa96b24e57ed168611d641a829cdefa2e7f690
Nov 01 10:32:09.596624 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:09.596608    4931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 10:32:10.286101 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:10.286056    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/must-gather-jdbxx" event={"ID":"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb","Type":"ContainerStarted","Data":"e805dd1b7937dbd1d8c850cd27fa96b24e57ed168611d641a829cdefa2e7f690"}
Nov 01 10:32:11.287271 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:11.287238    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:32:11.292904 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:11.292876    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:32:11.293850 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:11.293810    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/must-gather-jdbxx" event={"ID":"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb","Type":"ContainerStarted","Data":"02a5eca2a6f687b6987b13cc06be08f128e2e4fd49acd6185a19c40d296bad52"}
Nov 01 10:32:11.314431 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:11.314407    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jd869_03c05080-9769-4bf4-ac9a-39b2c7301170/console-operator/2.log"
Nov 01 10:32:11.318444 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:11.318421    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bj2zg_f13e4499-bb2f-4e37-bd37-52f4bdad02dd/ovn-acl-logging/0.log"
Nov 01 10:32:12.305911 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:12.305873    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/must-gather-jdbxx" event={"ID":"9ac5e364-d5b7-4d18-bd91-ac238c0fd0bb","Type":"ContainerStarted","Data":"5b439c36c97d62b53b426cb03c5b5e837248707a0bf249a44f8a0a3f465f2914"}
Nov 01 10:32:12.426793 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:12.426740    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w96kq/must-gather-jdbxx" podStartSLOduration=2.9756615650000002 podStartE2EDuration="4.426726382s" totalImagesPullingTime="1.451064817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:32:12.425706528 +0000 UTC m=+7202.540579495" watchObservedRunningTime="2026-11-01 10:32:12.426726382 +0000 UTC m=+7202.541599350"
Nov 01 10:32:19.647051 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:19.646623    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v7gkj_2e9c65da-47f6-494b-819e-34ee9896ff41/global-pull-secret-syncer/0.log"
Nov 01 10:32:20.271795 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:20.271669    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-67r69_4f63fd1a-4af7-414d-aa39-4580e4e98d7f/connection-checker/0.log"
Nov 01 10:32:20.544494 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:20.534707    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-z247t_cc0775c9-babb-41b8-8db2-f958a8f69c95/manager/0.log"
Nov 01 10:32:20.704738 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:20.704690    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fg7b2_46b41145-f5ea-4af3-9f72-b12fee0dd5e2/connection-checker/0.log"
Nov 01 10:32:21.141779 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:21.141684    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ftpbm_bcd6ef98-6c6a-4687-bc85-4a540e06022a/connection-checker/0.log"
Nov 01 10:32:21.555494 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:21.555379    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m97nw_c07de17d-572b-4a2c-a44c-b74048d7ec0a/manager/0.log"
Nov 01 10:32:21.877049 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:21.873273    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dssdt_8d282f7e-e082-4cc0-ac9b-39880957b56e/server/0.log"
Nov 01 10:32:22.496687 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:22.496652    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vlpgj_e5d1e97d-be9c-4f45-827c-19f789823a3b/konnectivity-agent/0.log"
Nov 01 10:32:23.765803 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:23.765626    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-244.us-west-2.compute.internal_36a0826bc10ea7285dbce782929bcb3b/haproxy/0.log"
Nov 01 10:32:24.489994 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:24.489961    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dcm8b_f5f45383-80fa-4ee1-9556-96ba0b54a988/proxy/0.log"
Nov 01 10:32:27.570706 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:27.570673    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-z247t_cc0775c9-babb-41b8-8db2-f958a8f69c95/manager/0.log"
Nov 01 10:32:28.520700 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.520660    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m97nw_c07de17d-572b-4a2c-a44c-b74048d7ec0a/manager/0.log"
Nov 01 10:32:28.794706 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.794631    4931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"]
Nov 01 10:32:28.802016 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.801992    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.843452 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.843421    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-sys\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.843452 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.843453    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-proc\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.843695 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.843467    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-podres\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.843695 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.843667    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-lib-modules\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.843795 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.843726    4931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpdm\" (UniqueName: \"kubernetes.io/projected/93ca6cc0-39da-40c7-bea8-75598fb1c870-kube-api-access-dnpdm\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.845828 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.844955    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6s9qr_ea75c58e-9f4a-4401-bcea-548e683abe2e/manager/0.log"
Nov 01 10:32:28.878160 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.878122    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"]
Nov 01 10:32:28.937487 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.937455    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dssdt_8d282f7e-e082-4cc0-ac9b-39880957b56e/server/0.log"
Nov 01 10:32:28.945096 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945062    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-lib-modules\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.945244 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945126    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpdm\" (UniqueName: \"kubernetes.io/projected/93ca6cc0-39da-40c7-bea8-75598fb1c870-kube-api-access-dnpdm\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.945244 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945178    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-lib-modules\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.945244 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945240    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-sys\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.945373 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945260    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-proc\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.945373 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945284    4931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-podres\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.945373 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945314    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-sys\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.945492 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945378    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-proc\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:28.945492 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:28.945427    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93ca6cc0-39da-40c7-bea8-75598fb1c870-podres\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:29.004655 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:29.004623    4931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpdm\" (UniqueName: \"kubernetes.io/projected/93ca6cc0-39da-40c7-bea8-75598fb1c870-kube-api-access-dnpdm\") pod \"perf-node-gather-daemonset-jqhkg\" (UID: \"93ca6cc0-39da-40c7-bea8-75598fb1c870\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:29.115730 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:29.115694    4931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:29.267488 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:29.267451    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zslv4_5cdcfa74-fd37-405c-8850-47dfa5b238e5/manager/1.log"
Nov 01 10:32:29.267488 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:29.267492    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zslv4_5cdcfa74-fd37-405c-8850-47dfa5b238e5/manager/0.log"
Nov 01 10:32:29.484980 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:29.484959    4931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"]
Nov 01 10:32:29.486369 ip-10-0-143-244 kubenswrapper[4931]: W1101 10:32:29.486337    4931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod93ca6cc0_39da_40c7_bea8_75598fb1c870.slice/crio-276c5b856c0e4b79a0d86066e35cd5a4cf879ee316be0f56b093d4cb46fee877 WatchSource:0}: Error finding container 276c5b856c0e4b79a0d86066e35cd5a4cf879ee316be0f56b093d4cb46fee877: Status 404 returned error can't find the container with id 276c5b856c0e4b79a0d86066e35cd5a4cf879ee316be0f56b093d4cb46fee877
Nov 01 10:32:30.098883 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:30.098846    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-svjt2_2e7e508b-b94f-4e2a-9bf1-c994ed627324/manager/0.log"
Nov 01 10:32:30.490823 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:30.490790    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg" event={"ID":"93ca6cc0-39da-40c7-bea8-75598fb1c870","Type":"ContainerStarted","Data":"f483f29d06551bd24e5f794204bb00202c39c022038a08079364edf43407d5a9"}
Nov 01 10:32:30.491006 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:30.490842    4931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg" event={"ID":"93ca6cc0-39da-40c7-bea8-75598fb1c870","Type":"ContainerStarted","Data":"276c5b856c0e4b79a0d86066e35cd5a4cf879ee316be0f56b093d4cb46fee877"}
Nov 01 10:32:30.491006 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:30.490901    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:30.605731 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:30.605665    4931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg" podStartSLOduration=2.605644953 podStartE2EDuration="2.605644953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 10:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:32:30.602235223 +0000 UTC m=+7220.717108192" watchObservedRunningTime="2026-11-01 10:32:30.605644953 +0000 UTC m=+7220.720517925"
Nov 01 10:32:31.025346 ip-10-0-143-244 kubenswrapper[4931]: E1101 10:32:31.025304    4931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4g77m" containerName="extractor"
Nov 01 10:32:31.507770 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:31.507743    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dcm8b_f5f45383-80fa-4ee1-9556-96ba0b54a988/proxy/0.log"
Nov 01 10:32:34.374152 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:34.374126    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-z247t_cc0775c9-babb-41b8-8db2-f958a8f69c95/manager/0.log"
Nov 01 10:32:35.241809 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:35.241776    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m97nw_c07de17d-572b-4a2c-a44c-b74048d7ec0a/manager/0.log"
Nov 01 10:32:35.670013 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:35.669987    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dssdt_8d282f7e-e082-4cc0-ac9b-39880957b56e/server/0.log"
Nov 01 10:32:36.510577 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:36.510549    4931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-jqhkg"
Nov 01 10:32:36.600919 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:36.600887    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-bksj6_f1f77890-a449-4958-9b83-ef93acbe2f58/cluster-monitoring-operator/0.log"
Nov 01 10:32:38.703372 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:38.703342    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-dcm8b_f5f45383-80fa-4ee1-9556-96ba0b54a988/proxy/0.log"
Nov 01 10:32:41.799764 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:41.799734    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-td2fk_5bdd19c7-7192-4fb7-bc9c-85438de618a3/node-exporter/0.log"
Nov 01 10:32:41.930786 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:41.930756    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-557c4f746fqdwtg_75d1c892-bf07-467c-92db-655a8d317413/istio-proxy/0.log"
Nov 01 10:32:42.220878 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:42.220844    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-td2fk_5bdd19c7-7192-4fb7-bc9c-85438de618a3/kube-rbac-proxy/0.log"
Nov 01 10:32:42.641124 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:42.641099    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-td2fk_5bdd19c7-7192-4fb7-bc9c-85438de618a3/init-textfile/0.log"
Nov 01 10:32:46.217034 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:46.216994    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-52vd7_f1ee5b01-a994-4efd-960e-93688cdfe984/istio-proxy/0.log"
Nov 01 10:32:46.424447 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:46.424419    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-5gxgq_ca5beb2f-1ccf-40e5-a862-e0264dbd35e4/istio-proxy/0.log"
Nov 01 10:32:47.937966 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:47.937935    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-rm7b4_60c755fb-ce57-44f7-9fe5-53879a8c8d1c/istio-proxy/0.log"
Nov 01 10:32:48.356132 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:48.356108    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-564f56d49f-s4j65_0b60d387-603f-4171-ac01-7a17b3dfddad/istio-proxy/0.log"
Nov 01 10:32:48.984667 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:48.984641    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6bf599bf-9cjkf_3ca1667a-32ed-4777-8755-ec7e72add722/router/0.log"
Nov 01 10:32:51.524788 ip-10-0-143-244 kubenswrapper[4931]: I1101 10:32:51.524762    4931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-557c4f746fqdwtg_75d1c892-bf07-467c-92db-655a8d317413/istio-proxy/0.log"